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ABSTRACT 


Due to growing operational constraints accelerated by the Global War on Terror, 
the United States Navy is looking for alternative methods of training to maintain its force 
in a high status of readiness. Updates in technology over the last decade have prompted 
Navy officials to take the initiative to update its training technologies. Computer-Based 
Instruction provides alternative means of training so that the training of war-fighters can 
be accomplished efficiently and effectively, saving the U.S. Navy time and resources 
while maintaining a high state of readiness. The goal of this thesis is to combine the 
principles of Instructional Design Systems (ISD) technology and advanced Simulation in 
order to produce a multimedia training capability for Navy and Marine Corps Officers. 

Specifically this thesis applies ISD principles to engineer and improve the Anti- 
Terrorism Force Protection (AT/FP) Officer Course Level II (ATO Level II Course) 
lessons for use by Navy and Marine Corps officers. The SavageStudio scenario 
generation application, originally designed for assessment of force protection measures in 
a port environment, is used to allow students to develop simulation lessons for this 
course. 

The final product of this thesis is a training course consisting of two lessons 
combining Computer- Based Training and Simulation providing interactive lessons and 
exercises, and two media delivery comparison studies with results for the Anti-Terrorism 
Force Protection lessons. The first study compared media effectiveness of Computer- 
Based Training versus Classroom instruction. The second study assessed the 
effectiveness of Computer-Based Training and the use of simulation for AT/FP. 
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I. INTRODUCTION 

A. PROBLEM STATEMENT 

The United States has been under terrorist threat and attack since the early 1980’s 
by ideological, political and religious groups that do not agree with United States policies 
and objectives around the world. According to sources from the Center for Security 
Forces (CENSEFOR 2006), the U.S. military has suffered more casualties due to 
terrorism than in combat operations during the period since the Beirut bombing attack in 
1983 to the beginning of Operation Iraqi Freedom. Terrorist groups recognize that the 
strength of the United States Armed Forces is unparalleled and therefore, it is impractical 
for these groups to challenge the U.S. military directly. Instead terrorist groups choose to 
attack vulnerabilities of U.S. forces in order to gain psychological, ideological and 
political support. 

Events such as the USS COEE attack on 12 October 2000, while refueling in 
Yemen, and the attempted missile attack on the USS ASHEAND and USS 
KEARSARGE, while moored in port in Jordan on 19 August 2005, reflect the Navy’s 
need for effective Eorce Protection employment of assets and Anti-Terrorism personnel 
training. It is in the Navy’s best interest that all hands be properly trained on Anti- 
Terrorism Eorce Protection (AT/EP) tactics techniques and procedures (TTP) to deter, 
defend, detect and mitigate against future attacks. 

While classroom instruction continues to be the most preferred delivery method 
of instruction, there has been a significant increase in technology usage over the past 
twenty years that enables other delivery methods of instruction. Classroom instruction is 
a cost-effective tool for small organizations but it can impose several constraints for large 
organizations such as time, cost and travel. While face-to-face training supports 
interaction and immediate feedback, learning can be achieved in other settings where the 
end user is not limited by time or location. Users can access the instructional lecture 
material whenever it meets their needs. 
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A large organization such as the United States Navy has its personnel deployed at 
locations all over the world, accomplishing a variety of different missions. Training costs 
are expensive. There are costs to maintain the classroom facilities, to pay the instructors, 
to maintain the teaching materials and aids, to provide travel and lodging for the sailors, 
and to make up for the work hours lost due to the sailors being away from work. 

Properly designed Computer-based Training (CBT), can address these man issues. 

B. BACKGROUND TRAINING METHODOLOGIES 

As a result of the Quadrennial Defense Review of 1997, the Department of 
Defense initiated the Advanced Distributed Learning (ADL) effort. The intent is to set 
forth a framework to provide DoD personnel the ability to achieve high quality education 
and training tailored to individual needs, and also to deliver cost effectively whenever 
and wherever is required (Executive Review of Navy Training Final Report, 2001). In 
2001 the Executive Review Board of Naval Training concluded that traditional classroom 
instruction was inefficient relative to cost and total time to train each student (Executive 
Review of Navy Training Final Report, 2001). The U.S. Navy commenced a review of its 
instructional methodology and implemented several new programs. 

One such new program was implemented at the Naval Air Technical Training 
Center in Pensacola Florida and the Submarine Eearning Center in Groton, Connecticut. 
The Apprentice Technical Training Program for Electronic Technical Training was 
created under the Naval Personnel Development Center (NPDC). Its purpose was to 
review and assess the electronic technician training curricula. Extensive reviews of the 
electronic training program were conducted by Subject Matter Experts (SME) as 
requested by the Nida Corporation, which was contracted to restructure, standardize and 
manage the implementation of the computer-based learning environment for the 
electronic oriented enlisted rates. The computer-based learning environment allowed 
students to proceed from basic training to the basic electronic school without having to 
wait for the class to be full enough to commence a new class. This allowed for the school 
administrators to schedule more classes, more often with fewer students. Students 
proceeded with lessons at their own pace and graduated ahead of their peers when 
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compared with the prior classes. Scenario-based testing was conducted and students 
demonstrated retention of proficiency levels in measuring, analyzing, and troubleshooting 
an array of previously studied circuits (Gulliver 2001). The Naval Training Facility 
located in Pensacola Florida was closed in 2004 during Hurricane Ivan but students were 
able to access the courses from Great Lakes without having to wait for the installation to 
fully restore all its classroom facility services. 

Another relevant Computer-Based Training (CBT) / Web-Based Trainer (WBT) 
training program was implemented in 2004 by John Ruffner, Kathleen Titley and Jim 
Fullbrook titled. Integrating Technologies for Shipboard Helicopter Signaling Skill 
Training. A prototype CBTAVBT of the Landing Signal Enlisted (LSE) course created by 
the DCS Corporation Orlando, Elorida was tested at the ESE School located in Naval Air 
Station (NAS) Norfolk. System developers and Subject Matter Experts (SMEs) reviewed 
current training curricula and Personnel Qualification Standards (PQS) to develop the 
course structure. The final product was created using the schoolhouse curriculum which 
matched more precisely the objectives of the instruction. Instructors at the ESE School 
expressed strong support for the resulting CBT. They commented that they planned to use 
the CBT course as a teaching aid to supplement classroom instruction and discussion as 
well as using it as a “study ahead” tool. The developers are working to formally evaluate 
and validate the CBTAVBT and also are working on a virtual-reality simulator, so that it 
can be included in the ESE schoolhouse curriculum (Ruffner, Titley and Eullbrook, 

2004). 

Updates in technology over the last decade have prompted the Chief of Naval 
Operations (CNO) to take the initiative to update training technologies for the United 
States Navy, as seen by various new technologies adopted for the training of naval forces. 
Of particular note is that the survival of our sailors depends on their ability to be properly 
trained on Anti-Terrorism Eorce Protection (AT/EP) use and employment of assets to 
deter, defend, detect and mitigate against future attacks. The creation of an AT/EP CBT 
with embedded simulation can improve training in such procedures. Research conducted 
by Sitmann, Kraiger, Stewart and Wisher (2006) and Hastings and Tracey (2005) points 
out the benefits of Computer-Based Training in comparison compared to other more 
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traditional methods of instruction. CBT and simulations can be effective and efficient, 
increasing learning and knowledge transfer without the constraints of classroom 
instruction. The benefits of a training program that can be accessed in any place at any 
time gives the sailors the ability to grow professionally and military. 

C. OVERVIEW 

The United States Navy has been looking at new technologies to provide training 
to its sailors. The evolution of computers and web applications over the last two decades 
has allowed the Navy to migrate to some degree prior classroom instruction into Web- 
based Instruction and Web-based portals such as Navy Knowledge Online (NKO). These 
new technologies offer some advantages that are necessary in today’s tempo of 
operations. Currently there are many Navy courses that can be restructured to allow for 
the delivery of the material using this Computer and Simulation-based technologies. 

The Mission of the Naval Education and Training Command (NETC) is to 
ensure fleet readiness and mission accomplishment, enhance professional 
and personal growth, and enable life-long learning. Guiding principles 
include: educate and train fleet requirements, provide the tools, learning 
and guidance that enable sailors to develop to their fullest potential, 
professionally and personally, provide responsive, best value solutions 
through rigorous analysis and disciplined, prioritized alignment with 
fleets, measure the success by the success of each sailor and the fleet, 
encourage innovation and personal initiative.(NETC 2007). 

As operational tempo increases based on the demands imposed by the Global War 
on Terror (GWOT), the United States Navy needs to continue investigating and applying 
technologies that allow sailors to be trained to their fullest potential with minimum 
impact to the Navy’s mission. 

D. MOTIVATION 

As a former Training Officer, this author has experienced first hand the rigorous 
training demands imposed on afloat and ashore commands from the increased operational 
tempo of the GWOT. Commands are required to achieve and maintain higher levels of 
readiness in all warfare areas; however the operational day remains twenty four hours 
long, with the same number of sailors and command budgets steadily decreasing. 
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In today’s Navy sailors no longer specialize in a single area. They must be trained 
to master multiple warfare areas such as Damage Control (DC), Visit Board Search and 
Seizure (VBSS) and Force Protection (AT/FP) in addition to their primary in-rate duties. 

It has been difficult for commands to decide when to allow sailors time away 
from the command to attend courses in a classroom setting, because the sailor may be 
needed for other ship operations allowing the sailor to be absent may decrease the 
command’s state of readiness. Decreasing budgets are also a factor when making these 
decisions because the command may not have the funds needed to send all sailors’s on 
Temporary Additional Duty (TAD) to a school for the required training. 

Computer-based Training (CBT) and Simulation-based Training (SBT) are 
underutilized technologies that can help in the training field. Modeling and Simulation 
technologies can be used to create Scenarios that exercise the individual and group 
learning skills. There have been previous attempts to use CBT and SBT to train sailors 
onboard ships. Some have been somewhat successful, such as Division Officer Training 
at Sea from the Surface Warfare Officer School (SWOS), Newport Rhode Island. Other 
training attempts have been less successful, such as the Navy’s General Military Training 
(GMT). Often the less-than-successful results are not due to the technology used to 
deliver the instruction, rather the way that the courses are organized and developed. If the 
material and content of a course is poorly developed, instruction results will be poor no 
matter what technology is used. The Instructional System Design (ISD) approach can be 
used to create, implement and evaluate course materials that can be later paired to the 
available technologies to achieve maximum knowledge gains. 

E. OBJECTIVES 

The objective of this thesis research is to develop and test a prototype instruction 
system to teach Navy, Marine Corps Officers and Senior Enlisted in AT/FP using the 
Instructional Systems Design (ISD) approach. The thesis tasks that need to be 
accomplished in order to meet this thesis purpose are: 
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1. Conduct a study into the Anti-Terrorism Officer (ATO) Level II course to 
analyze what training objectives are achievable, based on the material that needs to be 
taught and the learner’s current capabilities. 

2. Conduct a task analysis based on cognitive and skill tasks needed to define the 
learning objectives for the training material, based on the information gathered from 
SME’s and current course notes. 

3. Develop a training delivery format, instructional methods and course content 
organization. 

4. Review and evaluate different media technologies that allow the delivery of the 
course content, taking into consideration time, resources, budget constraints and course- 
learning goals. 

5. Conduct experiments that focus on determining the most effective training 
strategy. 

6. Report results and recommendations to Navy training commands in order to 
encourage further development and deployment of these important new tactical decision 
aids and technologies. 

F. THESIS ORGANIZATION 

The thesis is organized as follows. Chapter II reviews background technologies, 
the Instructional System Design and previous work that motivated this thesis. Chapter III 
provides and in-depth look at the Analysis, Design and Development phases for the 
creation of the CBT lessons. Chapter IV provides a detailed explanation of the 
methodology used in the implementation phase of the thesis project. Chapter V outlines a 
step-by-step instruction on how to use SavageStudio in the creation of a Force Protection 
Simulation Scenario. Chapter VI results of the two studies conducted are discussed in 
great detail. Chapter VII provides a summary of the conclusions from this work and 
provides recommendations for future work. Appendix A contains the task analysis for 
lesson development. Appendix contains lesson enabling objectives. Appendix C contains 
media selection criteria. Appendix D and Appendix E contain lesson storyboards. 
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Appendix F thru Appendix I contain lesson pre and post-tests. Appendix J contains 
sample port brief information sheet. Appendix K contain Institution Review Board (IRB) 
forms sample. Appendix L contains the student questionnaire. Appendix M contains the 
SavageStudio exercise task checklist. Appendix N contains an Analyst report sample. 
Appendix O contains supporting materials and information on how to request 
SavageStudio and AT/FP lesson DVDs. 
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II. RELATED WORK 


A. INTRODUCTION 

This chapter reviews prior Naval Postgraduate School (NPS) theses that served to 
motivate this thesis work. It also discusses the Anti-Terrorism Officer Level II course 
(ATO Level II course), the Scenario Authoring and Visualization for Advanced 
Graphical Environments (Savage) Studio simulation environment, and the Instructional 
Systems Design (ISD) concept used in the completion of this thesis. Relevant research 
about training effectiveness between Computer-Based Training (CBT) and Classroom 
Instruction (Cl) is also discussed. Additional information on the topics related to this 
thesis can be found in the references section at the conclusion of this thesis. 

B. ANTI-TERRORISM OFFICER LEVEL II COURSE (ATO LEVEL II 

COURSE) 

The purpose of the Anti-Terrorism Officer (ATO) course is to incorporate the 
Level II mandated training requirements contained in DODINST 2000.16 to provide the 
ATO with the necessary skills and knowledge of the six elements of the AT program: 
Planning, Risk Management, Training, Exercises, Resources and AT Program Review. 
Graduates are qualified to manage their command's anti-terrorism program ashore and 
afloat (CENSECEOR 2006). Eesson topics include the concept of defense in depth, 
development of pre-planned responses, duties and responsibilities of an ATO, use of 
force, introduction to terrorism and terrorism operations, AT resources and tools, tactic 
techniques and procedures (TTP) and AT plan development and procedures 
(CENSECEOR 2006). Training is accomplished through a combination of classroom 
lectures and practical scenarios relevant to both shore-based and afloat commands. 

The scope of the course is to train Navy commissioned officers and selected 
enlisted E-7 and above, USMC personnel and DoD equivalent civilians as Anti- 
Terrorism Officers (ATO). The course provides instruction in the U.S. Navy and Marine 
Corps Anti-Terrorism and Eorce Protection (AT/EP) programs as described in 
OPNAVINST 3300.55, MCO 3302.1C and DODINST 2000.16 (CENSECEOR 2006). 
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Course material includes the process for developing of comprehensive anti-terrorism 
plans, AT program management and AT program assessments and execution. The ATO 
Level II course is organized by classroom instruction followed by a practical scenario 
development. The course consists of five days of instructor led course instruction. The 
instructor uses presentation slides, class discussion and an assortment of visual aids 
during the initial three days of the course. 

Topics covered in the ATO Level II Training Course are presented in Figure 1. 


Unit I 

I-1-1 - Introduction 

I-2-1 - Introduction to Terrorism/Terrorist Operations 

1-3-1 - Introduction to Terrorism Program 

1-4-1 - (FOUO) Improvised Explosive Devices (lED's) 

1-5-1 - Use of Eorce/Deadly Eorce 

1- 6-1 - Anti-terrorism Officer Information and Tools 
Unit 2 

2- 1-1 - Defense in Depth/Preplanned Responses 

2-1-2 - Pre-Planned Responses 

2-2-1 - Eorce Protection Conditions (EPCON's) 

2-3-1 - Surveillance 

2- 4-1 - Individual Protective Measures 
Unit 3 

3- 1-1 - Anti-terrorism Risk Management 

3-2-1 - Anti-terrorism Threat Assessment and Threat Eevels 
Unit 4 
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4-1-1 - Introduction to Anti-terrorism Plans 

4-1-2 - Sample In-port Security Plan 

4-2-1 - Anti-terrorism Planning Proeess 

4-3-1 - Anti-terrorism Plan Format 

4-3-2 - JAT Guide Anti-terrorism Plan Basie Plan 

4-4-1 - AT Program Funding 

4-5-1 - USS Cole Case Study 

4-6-1 - Khobar Towers Case Study 

4-7-1 - Task One - Foree Proteetion Conditions Exereises 

4-7-2 - Post Orders for Main Gate and Industrial 

4-8-1 - In-port Seeurity Plan (ISP) Praetieal Applieation 

4-9-1 - Ashore Table Top Exereise Brief 

4-9-2 - Afloat Table Top Exereise Brief 

Eigure 1. The ATO Eevel II Training Course Outline ineludes a variety of AT/EP 

topics. 

On the fourth day students are divided into teams to develop a foree-proteetion 
plan for a given seenario. On the fifth day the student-prepared plans are presented by 
eaeh team leader. Eeedbaek is provided by the instructors and other students. 

C. RELEVANT NAVAL POSTGRADUATE SCHOOL THESES 

LT. James Harney’s thesis, Analyzing Anti-Terrorist Tactical Ejfectiveness of 
Picket Boats for Force Protection of Navy Ships Using X3D Graphics and Agent Based 
Simulation (Mareh 2003) deseribes the use of open souree, web, modeling and simulation 
technologies used in the development of a prototype to aid taetieal operators in the Anti- 
Terrorism and Eorce Protection planning. The prototype allows for the operator to ereate. 
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visualize and analyze Anti-Terrorism and Force Protection “what-if ’ scenarios. The 
operators use these hypothetical scenarios to aid in the development of an effective Force 
Protection plan. 

LT Patrick Sullivan’s thesis, Evaluating the Ejfectiveness of Waterside Security 
Alternatives for Eorce Protection of Navy Ships and Installations Using X3D Graphics 
and Agen-Based Simulation (September 2006) expands LT Harney’s thesis to include 
other force protection measures for Navy ships and installations. The prototype created 
by LT Harney was refined during LT Sullivan’s thesis to provide Naval Installation 
Security Planners, Harbor Operations Support Staff and shipboard Force Protection 
Officers the ability to evaluate security alternatives using simulations. Furthermore the 
Scenario Authoring and Visualization for Advanced Graphical Environments 
(SavageStudio) scenario tool provides Naval Installation Security Planners with the 
ability to model, simulate and analyze new systems being considered in order to assess 
overall risk reduction and return on investment. 

D. SCENARIO AUTHORING AND VISUALIZATION FOR ADVANCED 

GRAPHICAL ENVIRONMENTS (SAVAGE) STUDIO 

SavageStudio is a scenario-generation analysis and visualization tool that 
provides the user with the ability to create complete AT/FP simulations with minimal 
setup and design (Sullivan 2006). SavageStudio allows users to fully generate scenarios 
that can be played back and analyzed using discrete event simulation (DES) and 
Extensible 3D (X3D) graphics. The scenario generation is accomplished by leveraging 
the extensible markup language (XME) architecture of the Savage, X3D model archive 
and Viskit discrete event simulation to generate complex simulations for visualization 
and analysis. 

The SavageStudio scenario generation tool provides the user with scenario 
creation capabilities using the same type of drag-and-drop interfaces used in most Navy 
Information Technology for the 2L‘ century (IT-21) applications. The interface allows 
users to select models (locations, friendly assets, hostile assets, neutral assets, etc) that 
will be used in the scenario. During scenario configuration, the user can change physical 
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parameter values (maximum speed, cruising speed, vessel draft, range, altitude, etc.) for 
each model and define their behaviors (friendly patrol craft, hostile explosive-laden 
vessel, etc). Once the models and their behaviors have been selected and assigned, 
SavageStudio creates a Viskit assembly file and an X3D model of the scenario which 
corresponds to the assembly file. 

Viskit is a visual programming tool for modeling behaviors and tactics that allows 
designers to use the Event Graph (EG) methodology to represent processes and 
mathematical equations as discrete event models (DES). A sample event graph is 
provided in Eigure 2. Assembling multiple event graphs into a single simulation allows 
the user to construct tactical scenarios of intent and derive numerical analysis from their 
results. In DES the simulation time clock advances as events occur rather than running in 
constant blocks of time. This approach enables faster running times, more scenario 
replications, and more thorough analysis results. 

The Viskit Graphical User Interface (GUI) written in Java is used to implement 
the event graph methodology. The Simkit GUI allows for statistical analysis of a 
simulation. The Diskit, Java API (application programming interface) extends the 
SIMKIT libraries. The DISKIT libraries enable the use of the IEEE Distributed 
Interactive Simulation Protocol (DIS) allowing the animation of 3D objects in a virtual 
environments and communications between different simulations. The Viskit, GUI 
allows the user with event graph knowledge to create computer simulation leveraging 
Simkit and Diskit libraries. It is important to recognize that the user does not need to 
know how to write Java code, he just needs to know how create an event graph. The 
event graph produced using the Viskit GUI is then saved as XME documents. This XME 
documents are transformed into JAVA source code using JAXB implementing the 
SIMIKT and DISKIT libraries. 


13 




Figure 2. The above figure is an example of the run method in event graph notation 

(from Buss 2004) 

The 3D model of the scenario which corresponds to the assembly file created by 
SavageStudio is created in Extensible 3D Graphics X3D Standard (Brutzman, Daly 
2007). X3D graphics is an ISO standard used in web pages, database applications and 
visual simulations. For more information regarding these topics refer to LT. Sullivan’s 
thesis (Sullivan 2006). 

E. INSTRUCTIONAL SYSTEM DESIGN 

The Instructional System Design (ISD) approach is the guiding methodology used 
in the lesson development for this study. The goal of the ISD approach is to select and 
use the correct instructional system to deliver the required knowledge to the learner. The 
development of the instructional system is structured in 3 parts: input, process and output. 
(Ford 1996) 

The input portion is for the ISD is intended to help understand the differences in 
the learner’s experience levels, the learner’s current level of knowledge about the subject, 
and the motivations of the learner. This information can be summarized as the learner’s 
background coupled with the desired goal of the instruction. The process portion is for 
the ISD develops the course and the media that delivers the instruction to the learner. The 
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output portion for the ISD is used to evaluate the effectiveness of the training to ensure 
that the system meets the training goals. Figure 3 contains the ISD concept map by 
Donald Clark. 



Figure 3. Instructional System Design (ISD) design concept map by Donald Clark 
http://www.nwlink.com/~donclark/hrd/sat2.html (1995). 

There are more than 100 different ISD models (Kruse 2007) but almost all are 
based on the “ADDIE” model. The ADDIE model, which stands for Analysis, Design, 
Development, Implementation and Evaluation, is used by the training community to 
standardize instruction and to impose consistent quality standards (Ciavarelli 2007). An 
ADDIE model as described by Steven J. McGriff is depicted in Eigure 4. A major benefit 
of using the ADDIE model is that each step’s outcome feeds the subsequent steps, 
ensuring that developers don’t proceed to subsequent steps without first completing the 
previous necessary step. 
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The ADDIE Mode) 
is an iterative 
instructional design 
process, where the 
results of the 
formative evaluation 
of each phase may 
lead the instructional 
designer back to any 
previous phase 

The end product of 
one phase is the 
starting product of 
the next phase. 



Figure 4. ADDIE model as described by Steven J. McGriff, Instructional Systems, 

College of Education, Penn State University (2000). 


The ADDIE model steps are briefly summarized in the following paragraphs. 

In the Analysis phase the problem needs to be defined using a job task analysis, 
determining what training is needed and why it is important. Developers identify the 
possible solutions to the problem, using task analysis, developers can explicitly define 
roles, tasks and outcomes of the targeted training. Developers define a sequence of tasks 
coupled with the required equipment that facilitate the execution of the task and measures 
of job performance are provided. The target audience has to be identified, developers 
need to analyze their knowledge and skills in order to determine what they need to 
complete the training and achieve satisfactory performance. The outputs of this phase are 
the inputs for the Design phase. 

The Design phase uses the outputs from the Analysis phase to plan a strategy and 
build a blueprint for the training program. Eirst the scope of the project is defined; writing 
objectives course content test and evaluation, delivery system and instructional strategies 
and sequence of instruction and resources. 

The Development phase uses the outputs of the design phase to generate lesson 
plans, student and teacher guides and the instructional materials required to complete the 
lessons. Here the actual training program is constructed using courseware authoring and 
software as specified in the design (Ciavarelli 2007). 
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The Implementation phase has two parts. The pilot phase creates or designs a pilot 
study and then delivers the instruction to a small group of subjects to test the efficacy of 
the training instruction. After the pilot study is completed, the second part is to revise the 
prototype and then deliver the overall training solution to the end-user population. 

The Evaluation phase measures the overall effectiveness of the training. Even 
though the evaluation phase is the last phase of the ADDIE model, developers are 
constantly evaluating the system to correct or improve the instruction before the final 
implementation. Thus evaluation takes place in all the phases of ISD (Ciavarelli 2007). 
The final version of evaluation determines whether the training program achieves the 
desired results and meets the proposed objectives. 

F. COMPUTER-BASED TRAINING (CBT) VERSUS CLASSROOM 

INSTRUCTION (Cl) EFFECTIVENESS 

The literature on CBT effectiveness reveals mixed results. Individual studies 
provide examples of CBT performing better, worse or having no effect. A 1992 IBM 
study demonstrated an up to 70 percent decrease in training time, with learning gains 
over 50 percent (Whitehouse & Pellegrin, 1995). Coppola and Myre (2002) addressed 
several previous studies that claimed CBT was not as effective as Classroom Instruction 
(Cl). In Wilson, Majsterek and Simmons (1996) four elementary students with learning 
disabilities performed worse with ‘Mathblaster’ CBT than with flashcards in a Cl setting. 
In Lee (1994) twenty six teachers did not learn well when they where required to use a 
CBT program to teach BASIC. Coppola and Myre cited those studies as examples to 
demonstrate alternate hypotheses that must be controlled for such as computer skills and 
attitude towards the material. 

When multiple studies are evaluated together, those cases that show significant 
differences in performance between media appear to be outliers. They can be explained 
by alternate hypotheses, or used evaluation samples that are not representative of the 
target population. Chumley-Jones, Dobbie and Alford (2002) reviewed 76 articles from 
1966 to 2002 about medical, dental and nursing CBT. They observed that studies with 
control groups where the educational content was constant demonstrated no difference in 
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knowledge gains. A meta-analysis of 96 studies from 1996 to 2005 reached a similar 
conclusion. Sitmann, Kraiger, Stewart and Wisher (2006) evaluated studies of adults 18 
years or older comparing WBL and Cl for job-related training programs. Training times 
ranged from one to 120 days. Only studies that provided enough data to allow for the 
calculation of a d statistic, or where the author provided such data when requested, were 
used. The various aspects of each study, such as response time, same or different method, 
experiment or case study etc., were then converted to a normalized scale by two raters 
with a third rater breaking any ties. They then looked at a subset of the studies that used a 
true experimental design, a subset that used equal instructional methods, and a subset that 
looked at using CBT in addition to CL They further separated instruction of declarative 
or procedural knowledge within each of the main subsets. Results are summarized as 
follows. 

The overall results indicated WBI was six percent more effective then Cl 
for teaching declarative knowledge, the two delivery media were equally 
effective for teaching procedural knowledge. However, WBI and Cl were 
equally effective for teaching declarative knowledge when the same 
instructional methods were used to deliver both WBI and Cl, suggesting 
media effects are spurious and supporting Clarks (1983, 1994) theory (p. 

623). 

Clark’s theory is best demonstrated by his often-cited truck analogy. “The best 
current evidence is that media are mere vehicles that deliver instruction but do not 
influence student achievement any more than the truck that delivers our groceries causes 
changes in our nutrition (Clark, 1983, p. 445).” 

Hastings and Tracey (2005) evaluate the articles and counter articles written by 
Clark in 1983 and 1994 and Kozma in 1991 and 1994. “His [Clark’s] position has always 
been that any given method can be developed for delivery by more than one medium. In 
1983 and even 1994, Clark’s argument was valid. In 2005 it is valid for many, but not 
all, instructional methods and delivery medium pairings. We assert that today, computers 
have unique, non replicable capabilities and therefore can support instructional methods 
that other media cannot” (p. 29). Hastings and Tracey purport advances in computers and 
the Internet over the last 22 years have transformed Clark’s computer-as-a-truck into a 

supersonic jet. As an example, in 1991 Kozma mentions that due to size and weight one 
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is more likely to learn from a book than a computer while riding a bus, but that may 
change with advances in technology. Kozma asserts computers have unique qualities that 
if utilized properly, can make CBT more effective than CL 

Greitzer (2002) describes how content developed for CBT was adapted for Cl 
instruction, thus improving both. The results reported by Sitzmann et al. (2006) and 
Schlager (1994) are directly applicable to the U.S. Navy because they only looked at true 
experiments on adults with at least basic computer and learning skills. There are many 
case studies, quasi-experiments and experiments to support claiming an advantage of 
CBT or CL When one looks only at studies in which true experimental methods were 
used, comparing the same material given to subjects similar to the population of the U.S. 
Navy, no difference was found between CBT and CL 

G. SUMMARY 

This chapter reviews prior Naval Postgraduate School (NPS) theses that served to 
motivate this thesis work. It also discusses the Anti-Terrorism Officer Level II course 
(ATO Level II course), the Scenario Authoring and Visualization for Advanced 
Graphical Environments (Savage) Studio simulation environment and the Instructional 
Systems Design (ISD) concept used in the completion of this thesis. Relevant research 
about training effectiveness between Computer-Based Training (CBT) and Classroom 
Instruction (Cl) is discussed. Additional information on the topics related to this thesis 
can be found in the references section at the conclusion of this thesis. Further information 
and greater in-depth knowledge of the subjects’ is provided in the list of references. 
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III. ANALYSIS, DESIGN AND DEVELOPMENT 


A. INTRODUCTION 

This chapter provides an in-depth look at the analysis, design and development of 
the lessons implemented using the CBT and SavageStudio scenario generation tool. The 
use of the instructional design model helps to ensure that the instruction developer makes 
important choices about the course structure and methods, thus ensuring that the training 
objectives are achieved. 

B. ANALYSIS 

I. Needs Analysis 

With the revolution in Training (Executive Review of Navy Training 2001) the 
United States Navy has been looking at new or alternative technologies to provide 
training to its sailors. The U.S. Navy is restructuring some of its courses to allow for the 
delivery of the material using Computer and Simulation-based technologies. During the 
last decade, the terrorist threat has been increasing as evidenced by terrorist attacks 
against U.S. forces around the world. There is a great need to train Naval and Marine 
Corps Officers and Senior Enlisted in the concepts of AT/EP. As operational tempo 
imposed by the GWOT increases, instructors need to look to other to look into 
technologies that allow sailors to be trained to their fullest potential anywhere at any 
time. 

Clearly the Navy’s needs are: 

1. Efficient or effective technologies that encourage faster learning and efficiency 
implementing the concepts. 

2. Training delivery both ashore and afloat using established infrastructure. 

2. Task Analysis 

Eormal task analysis helps the developer determine what specific tasks need to be 
accomplished in order to fulfill each learning objective. Tasks are broken down in 
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subtasks, and each subtask is matched to a learning objective. The task analysis also helps 
developers build performance measures that help to determine student success or failure 
when performing the task. A detailed task analysis supporting the objectives of this thesis 
is provided in Appendix A. 

3. Audience 

A thorough understanding of the participants is essential for developing 
successful instructional material. There is a need to develop an audience profile to ensure 
that the instructional design is compatible with the students’ skill and knowledge level. 
The lessons in this course are intended for Officers and Senior Enlisted assigned to 
AT/FP duties. The group of Officers is comprised of all Naval Officers, all Marine Corps 
Officers, Limited Duty Officers in the military police rates and Senior Enlisted in the Fire 
Control, Gunners mates and Master at Arms rates. Given the nature of the military rank 
structure, most Limited Duty Officers and Senior Enlisted in the mentioned rates often 
possess some post-secondary college education or degree. This indicates that Officers and 
Senior Enlisted continued their education during the course of their military service. 

Commissioned Officers are required to have a college degree. With the 
implementation of IT-21 and the Navy Marine Corps Internet (NMCI) and the 
educational background of the audience, it can be assume that they have at least basic 
computing skills necessary for the applications used in this research. The level of 
enthusiasm is high among the students selected to attend the course, and they are usually 
selected to attend the course due to high level of confidence by their superiors. 

The background knowledge that the students have in the Force Protection subject 
varies, from basic to advanced. Some Naval Officers ranks 0-1 thru 0-3 have basic 
knowledge, while the Limited Duty Officers and the Senior Enlisted have more advanced 
knowledge and have been exposed to the subject since some have served in the AT/FP 
rates. 
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C. DESIGN 

1. Learning Objectives 

The learning objectives are the center of each lesson design and help describe the 
knowledge, skills and attitudes (KSAs) expected of the student after completion of the 
lesson activities. By meeting the learning objectives the student will be able to achieve 
the course goal. The objectives clearly state what the student is expected to know 
(knowledge) or perform (skill) at the end of each lesson, and to what degree in order to 
properly demonstrate successful completion of the objective. 

The overall student learning objectives for the ATO course are defined as: 

1. Manage the overall Anti-terrorism Program as the Commanding 
Officer's primary advisor for a shore-based or afloat command, including personnel, 
physical resources, and training requirements. 

2. Develop and implement Pre-Planned Responses that will be effective 
mechanisms to detect, deter, defend and mitigate likely terrorist threats against U.S. Navy 
assets and installations. 

3. Assess the anti-terrorism environment, situation and unit readiness for 
an ashore or afloat command, given current contingencies and sources of information. 

4. Develop a comprehensive Anti-terrorism Plan incorporating anti¬ 
terrorism principles, concepts, program elements and standards across the range of 
military operations, at any echelon of command both afloat and ashore, utilizing the AT 
planning process. 

The learning objectives for the created lesson are defined as: 

1. Define the importance of the Defense in Depth concept in accordance 
with Navy AT/FP Tactics, Techniques and Procedures (TTP). 

2. Apply the procedures of the Defense in Depth concept to a ship’s in¬ 
port preparations. 


23 



3. Assess the anti-terrorism environment and situation to identify port 

vulnerabilities. 

4. Define and assess the anti-terrorism environment and situation to 
identify critical assets and vulnerabilities for a ship in-port. 

5. Assess the anti-terrorism environment and situation to develop a plan 
that mitigates vulnerabilities to an acceptable risk for a ship in-port. 

6. Assess the anti-terrorism environment and situation to assess risk to a 
ship in-port and develop appropriate countermeasures. 

7. Define and assess the anti-terrorism environment and situation to 
identify and quantify the threat. 

8. Define and assess the anti-terrorism environment and situation by 
identifying critical assets and vulnerabilities and discuss force protection employment to 
protect these assets. 

9. Define the assessment zone as presented in the Defense in Depth 
concept and in accordance with Navy AT/FP TTP. 

10. Demonstrate the application of the assessment zone as described in the 
Defense in Depth concept and in accordance with Navy AT/FP TTP. 

11. Define the warning zone as described in the Defense in Depth concept 
and in accordance with Navy AT/FP TTP. 

12. Demonstrate the application of the warning zone as described in the 
Defense in Depth concept and in accordance with Navy AT/FP TTP. 

13. Define the threat zone as described in the Defense in Depth concept 
and in accordance with Navy AT/FP TTP. 

14. Demonstrate the application of the threat zone as described in the 
Defense in Depth concept and in accordance with Navy AT/FP TTP. 

15. Define the purpose of pre-planned responses in accordance with Navy 
AT/FP TTP and NTTP 3-07.2.1 (Rev A). 
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16. Define the likely terrorist threats by applying the pre-planned 
responses concept in accordance with Navy AT/FP TTP. 

2. Performance Standards 

Upon completion of the lesson the student will be able to: 

1. Successfully define the importance of the Defense in Depth concept in 
accordance with Navy AT/FP TTP. 

2. Successfully identify the definition among other distracters of Defense 
in Depth concept in accordance with Navy AT/FP TTP. 

3. Successfully apply the procedures of the Defense in Depth concept to 
ship in-port security scenario by: 

a. Correctly identify port vulnerabilities. 

b. Correctly identify own high value assets that need protection. 

c. Correctly identify own critical assets vulnerabilities. 

d. Mitigate vulnerabilities to an acceptable risk. 

e. Correctly establish a warning zone. 

f. Correctly establish a threat zone. 

g. Correctly establish an assessment zone. 

h. Correctly employ force protection assets to protect high value 
assets. 

i. Assessing risk. 

j. Develop countermeasures. 

k. Identify and quantify the threat. 

4. Successfully assess the anti-terrorism environment and situation given 
intelligence reports and visual aids. 


25 



5. Correctly define the purpose of pre-planned responses in accordance 
with Navy AT/FP TTP and NTTP 3-07.2.1 (Rev A). 

6. Successfully identify the definition among other distracters of the 
purpose of pre-planned responses in accordance with Navy AT/FP TTP and NTTP 3- 
07.2.1 (Rev A). 

7. Complete an examination with an 80% correct answer rate of the 
following terms: 

a. Layered defense concept definition and application. 

b. Assessment zone definition. 

c. Warning zone definition. 

d. Threat zone definition. 

e. Define the purpose of the assessment, warning and threat zones. 

f. Organization of the Naval Vessel Protection zone. 

g. Assets used in the assessment, warning and threat zones. 

h. Watch-stander actions in the assessment, warning and threat zones. 

i. Watch-stander determination of threat. 

j. Definition and purpose of pre-planned responses. 

k. Identify future threats as defined by Navy instructions. 

3. Enabling Objectives 

The enabling objectives determine the pre requisite skills required for completion 
of each specific learning objective. Enabling objectives define the execution of the 
required tasks that accomplish each specific learning objective. Enabling objectives for 
the lesson prepared in this thesis are detailed in Appendix B. 
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4. 


Media Selection 


The learning objectives can be presented to the student using several teaching 
strategies, including lecture and practical exercises. The teaching strategy that can be 
used can be either classroom lecture or CBT lecture with simulation. The media selection 
used has to be based on the needs of the Navy and the technology that allows the proper 
implementation of the learning objectives. A detailed explanation of the methodology 
used for media selection can be found in Appendix C. 

To determine which media can be used for a Navy-wide prototype, the following 
questions need to be assessed: 

1. How many people need the training? 

2. Is classroom instruction viable and cost effective? 

3. Can all the target audience participants attend the training? 

4. Do the participants need the instruction as refresher training? 

5. Do the participants need immediate feedback from subject matter experts? 

6. Do participants have adequate technology to perform the training? 

7. Will the training be self- paced or sequenced by classroom schedules? 

After assessing the Navy’s current and future capabilities it has been determined 
that the lessons can be viewed using any current Navy Desktop workstation with the 
following minimum requirements: 

Operating system: Windows, Macintosh or Linux 

Video Cards: Standard 

Memory 512 MB Single Channel DDR2 SDRAM 

Hard Drive: 60 GB Hard Drive (7200 RPM) 

Monitor: Standard 

Optical Drive: 24X CD ROM 

Sound: Standard speakers or standard headphones 
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Mouse: Standard, ergonomic design 


5. Resources 

The resources used for the development of the lessons were provided by the 
Center for Security Forces (CENSECFOR) 1575 Gator Boulevard, Suite 226, Norfolk, 
Virginia 23521-2751. Additional ressources for the X3D content of the course were 
provided by the SAVAGE research group in the MOVES Institute at the Naval 
Postgraduate School. 

D. DEVELOPMENT 

Eessons were constructed using the instructional material currently used in the 
ATO Eevel II course provided by CENSECEOR. Due to the experimental subjects’ time 
constraints and availability the lessons used for this thesis were modified from those used 
in the ATO Eevel II course to accommodate for these constraints and to allow for the 
proper completion of the tested lessons. After the task analysis was completed, scenario 
storyboards (detailed storyboards are located in Appendix D and Appendix E) were 
created for the two studies that are covered in this thesis. During media selection it was 
determined that for the delivery of the knowledge objectives of the course is used 
Articulate Presenter trial version software and use the SavageStudio for the delivery and 
performance assessment of the skill objectives in the study. 

Articulate Presenter trial version ( www.articulate.com) is a Plash authoring tool 
that was used for the transformation of the storyboards into Plash media. The software 
allows the developer to add visual graphics, notes and audio to the storyboards creating 
interactive movies. Por a full tour, demos and an overview of the Articulate software are 
at ( http://www.articulate.com/products/presenter.php) . 

SavageStudio was used to develop the scenarios used for the skill objectives 
performance and assessment. SavageStudio was used in the scenario generation allowing 
the user to create complete simulations with minimal setup. SavageStudio provided the 
scenario creator (student) with drag and drop interfaces for minimum preparation on how 
to use the software. 
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The created prototype used for data collection employs a mixed-mode approach 
where the instructor uses both CBT and X3D graphics for instruction and scenario 
development. The subject matter expert is available during the instruction to assist and 
answers any student questions. 

1. Instructional Materials 

The storyboard created for the CBT vs. Cl effectiveness comparison (Lesson 1) is 
located in Appendix D 

The storyboard created for the CBT effectiveness (Lesson 2.1) is located in 
Appendix E. 

E. SUMMARY 

This chapter provides an in-depth look at the analysis (task analysis and target 
audience), design (learning objectives, performance standards, enabling objectives, media 
selection, and resources) and development of the developed lessons for the AT/FP course 
using the ISD framework. These lessons can be used for the AT/FP course in conjunction 
with the SavageStudio scenario generation tool. The ADDIE model was used to ensure 
that the course objectives meet the ATO Fevel II course objectives. This methodology is 
advisable and suitable for other courses. 
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IV. METHOD 


A. INTRODUCTION 

This chapter provides an in-depth look at the two studies conducted during this 
thesis research. The chapter is divided in the following manner: study I AT/FP CBT 
versus Cl, and Study II Training effectiveness of the ATFP CBT with Simulation. The 
participants, variables measured, apparatus and procedures used in the implementation 
are presented. 


B. STUDY I 

I. Participants 

The pool of volunteer participants for study I was comprised of 16 students of the 
OA 3402 course and other MOVES students attending the Naval Postgraduate School. 
The students of the course are Navy and Marine Corps and officers ranks range from 0-2 
to 0-4 male and female. 


Study 1 Participant's Information 

RANK 

RATE 

GENDER 

AGE 

SERVICE Yrs, Mo. 

0-3 

USN, SWO 

F 

34 

10 

0-4 

USN, PILOT 

M 

35 

13.9 

0-2 

USN, HR 

M 

25 

3.6 

0-3 

USN, SWO 

M 

34 

16.2 

0-3 

USN, SWO 

M 

27 

4.9 

0-4 

USN, PILOT 

M 

33 

11 

0-3 

USN, SWO 

M 

32 

14 

0-4 

GERMAN, ARMY 

M 

27 

6 

0-4 

USN, IP 

M 

34 

15 

0-3 

USN, IP 

M 

34 

14.6 

N/A 

CIV 

M 

35 

10 

N/A 

CIV 

M 

33 

10 

0-4 

GERMAN, NAVY 

M 

32 

12 

0-3 

USN, HR 

M 

31 

8 

0-3 

USN, HR 

F 

29 

7 


Table 1. Study I Participant’s information. 
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The participant’s previous assignments include Navy ships, Navy submarines, 
shore installations, and aviation commands ashore and afloat. The treatment of subjects 
was in accordance with the ethical standards of the American Psychological Association 
(APA). 


2. Experimental Design 

The independent variable for this experiment is the mode of delivery, media, of 
instruction. Two conditions of the independent variable are considered, CBT and CL The 
dependent variable was post-test scores. Scores were taken after completion of the 
training. Demographics of interest include: age, gender, military branch, type and number 
of past duty assignments (jobs), education level, learning style and computer skills. 

The purpose of study I, was to determine if there is a difference in performance 
between groups that received lessons with the same content using different media 
delivery, namely CBT and CL Answering this question is a first step in determining if 
CBT is a viable, cost-effective and flexible alternative to Cl training for AT/FP training 
requirements. Study details follow. 

A. All assessments where done using printed material. 

B. The following experimental design was used to compare CBT vs. Cl media 
effectiveness. Participants received AT/FP training using different media delivery 
methods Cl, CBT. Pre- and post-tests were conducted to examine mean differences 
between groups after training was completed. Table 2 shows the experimental design for 
Study 1. 
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Instructional Pre-Test Treatment Post-Test 

Media 


CBT 

Xo 

Oi 

Xi 

Cl 

Xo 

O 2 

Xi 


Table 2. Study I Experimental Design. 0i refers to CBT treatment and O 2 refers to Cl 

treatment. 

Null hypothesis: Mean score of post-test for the CBT group equals the Mean 
score for the post-test of the Cl group. 

Alternate hypothesis: Mean score of post-test for the CBT group does not equal 
the Mean score for the post-test of the Cl group. 

3. Apparatus 

The classroom training, used the NPS Human Systems Interaction and Learning 
(HSIL) lab, 1 Dell Precision M90 computer 2.33GHZ with 2.00G RAM that contained 
the slides created by the subject matter expert (SME). The power point slides were 
created using learning and enabling objectives used in the Anti-terrorism Officer Course 
Level II. The operating system for this computer is Windows XP Professional Version 
2002 with Microsoft Power Point version 2003. The projector is Sony VPL-XIOOO with a 
resolution of 1024 X 768, 1100 lumens. 

The CBT portion of the research used each student’s personal laptop. The pre-test 
and post-test were done as part of the CBT. The CBT will be created by the SME using 
the ‘Articulate’ presenter professional software, using the same learning and enabling 
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objectives used in the creation of the Power Point slides. The CBT used the same images 
and text as the Cl power point and included a voice narration of the information spoken 
by the instructor in the CL 

4. Procedure 

Participants were assigned at random to the classroom instruction or to the CBT 
instruction groups depending on the last digit of their social security number. Participants 
assigned to the classroom instruction group were first briefed orally and in writing by the 
subject matter expert on the Institutional Review Board (IRB) process and then asked to 
complete the student background surveys. After completion of the surveys, the students 
were instructed to complete a pre-test on the AT/FP topic of instruction. Upon 
completion of the pre-test, students were instructed to sit in a classroom where classroom 
instruction on the AT/FP topic was conducted using an SME (this author) as a facilitator. 
After completion of the classroom instruction, students were instructed to conduct a post¬ 
test on the AT/FP topic of instruction. 

Participants assigned to the CBT instruction group were briefed on the 
Institutional Review (IRB) process and asked to complete the student surveys. After 
completion of the surveys, the students were instructed to complete a CBT embedded 
pre-test on the ATFP topic of instruction. Upon completion of the pre-test, students were 
instructed to proceed with the CBT lesson. After completion of the CBT lesson students 
were instructed by the CBT to conduct a CBT embedded post-test on the AT/FP topic of 
instruction. 

C. STUDY II 

I. Participants 

The pool of volunteer participants for study II is comprised of 20 Officers (2 
females and 18 males) of the following Naval and Marine Corps communities: Surface 
Warfare Officers (SWO), Information Professionals (IP), Meteorological Officers 
(METOC), Communications Officers (COMMO, USMC), Pilots, Naval Plight Officers 
(NPO), Intelligence Officers (IW), Human Resources (HR) and Medical Officers (MD), 
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all attending the Naval Postgraduate Sehool. The participant’s previous assignments 
include Navy ships, Navy submarines, Navy and Marine Corps shore installations and 
aviation commands ashore and afloat. The participants age range from 26 years old to 42 
years old with an average age of 33 years. The participants’ rank from LT/0-3 USN or 
Capt./0-3 USMC to CDR 0-5 USN. The participants’ total military service time range 
from 5 years to 19 years of military service time, with an average military service time of 
12.63 years. The participants reported that 14 had attended AT/FP training courses and 6 
had not attended AT/FP courses. The participants reported the following regarding their 
perceived knowledge of AT/FP: 3 participants report no AT/FP knowledge, 3 participants 
report less than basic AT/FP knowledge, 11 participants report having at least basic 
AT/FP knowledge, 3 participants report having reasonable knowledge (more that the 
basic knowledge) of AT/FP, and no participant reported having expert knowledge of 
AT/FP. Participants also reported no previous AT/FP training in the last 12 months while 
in their present command. Most participants reported to having AT/FP training once a 
year in their previous command, however some reported AT/FP training weekly, monthly 
and semi-annually in their previous commands. The treatment of subjects was in 
accordance with the ethical standards of the APA. 
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Study II Participant's information 


Rank 

Rate 

Gender 

Age 

Total Time 
in the 

Military 

Previous 

Knowledge 

0-4 

Commo 

male 

39 

12.3 

0 

0-3 

IP 

male 

29 

6 

1 

0-3 

Sub 

male 

36 

17.05 

1 

0-5 

Human 

Resources 

female 

42 

18.85 

1 

0-4 

NFO 

male 

34 

13.5 

1 

0-3 

Metoc 

male 

29 

8.15 

1 

0-3 

Metoc 

male 

37 

18.45 

1 

0-4 

SWO 

male 

37 

18 

0 

0-4 

Doc 

male 

34 

17 

1 

0-3 

Commo 

male 

34 

15. 8 

1 

0-3 

Metoc 

female 

32 

10 

1 

0-3 

Intelligence 

male 

35 

16 

1 

0-3 

SWO 

male 

27 

6 

1 

0-3 

IP 

male 

34 

14 

1 

0-3 

Pilot 

male 

31 

8.67 

0 

0-3 

Intelligence 

male 

35 

6 

0 

0-3 

IP 

male 

26 

5 

0 

0-3 

Commo 

male 

31 

14 

1 

0-3 

SWO 

male 

33 

16 

1 

0-3 

IP 

male 

37 

15 

0 


Table 3. Study II Participant’s information. 


2. Experimental Design 

The purpose of study II is to determine the effectiveness of the ATFP instruction, 
namely the improved performance of students after completing the ATFP lesson and 
simulation). Answering this question is a first step in determining whether future studies 
on AT/FP training and simulation can be used to compare against classroom instruction. 

The independent variable for this experiment is the treatment, namely combined 
AT/FP lesson and simulation-based training. The dependent variable for this experiment 
was post-test scores, namely, individual knowledge of each individual on the AF/FP topic 
(cognitive) after lesson completion. Another variable considered is the number of 
successfully completed tasks for a particular scenario using SavageStudio. Scores were 
taken during task execution and evaluation was performed after completion of each 
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scenario. Demographics of interest include: age, gender, military branch, type and 
number of past duty assignments (jobs), education level and computer skills. 

A. All assessments were done using printed material. 

B. The following experimental design was used to compare the effectiveness of 
the CBT with simulation. Participants received AT/FP training using CBT with 
simulation. Pre and post-tests were both given to examine learning effects. Table 4 shows 
the study II experimental design parameters of interest 


Instructional Observation Observation 

Media 


CBT 

PRE-TEST 

POST-TEST 

SIMUEATION 




Table 4. Study II Experimental Design. 


Null hypothesis: mean score of pre-test equals the mean score of post-test. 

Alternate hypothesis: mean score of pre-test does not equal the mean score of 
post-test. 


3. Apparatus 

For the learning portion of the study, the researchers set up rooms at the NPS 
library with 1 Dell Precision M90 computer 2.33GHZ with 2.00G RAM that contained 
the CBT lessons created by the subject matter expert (SME) using the Articulate trial 
software. The computer provided to the students also contained the SavageStudio 
software. The SavageStudio software can be downloaded from the Scenario Authoring 
and Visualization for Advanced Graphical Environments website located at 
https://savage.nps.edu/Savage/ , description on scenario creation using SavageStudio are 
provided in greater detail in Chapter V. 
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The operating system for this computer is Windows XP Professional Version 2002 with 
Microsoft Power Point version 2003. For completion of the IRB forms and student 
questionnaires the participants used paper forms. 

4. Procedure 

The students (30 total) received e-mail requests to participate in this study. Of the 
30 students, 25 replied back with interest on participating, and of those 25 only 20 were 
able to complete the tasks due to time availability and conflicts in schedules. Students 
were instructed to report to the NPS library at designated times for the study. Upon 
reporting to the NPS library, each student was briefed on the Institutional Review process 
(IRB) and on the experimental procedures, written and referenced in Appendix J thru 
Appendix N. Students were asked to complete the student survey. After completion of the 
surveys, the students were instructed to complete a pre-test on the ATFP topic of 
instruction. Upon completion of the pre-test, students were instructed to complete the 
CBT instruction. After completion of the CBT, students were instructed to contact the 
Subject Matter Expert (SME) for further instructions. The SME directed the student to 
read the scenario information sheet. The SME directed the student with detailed scenario 
generation and execution instructions and then asked the student to use SavageStudio to 
generate and execute a scenario. The following paragraphs provide a summary of the 
student required actions, detailed scenario generation and execution instructions are 
provided in Chapter V. During the scenario generation, the student was asked to select a 
location where the scenario took place. Student then selected the assigned friendly assets 
HVU, patrol boxes, patrol boats, radars and barriers. Once friendly assets were selected, 
the student placed the assets in a location where they were more secure, based on critical 
vulnerabilities of the location and asset capabilities. The student was asked to define the 
assessment, warning and threat zones by using the assets provided namely radars and 
patrol boxes. Students placed and modified patrol boxes where the location of the patrol 
box would increase the probability of the intercept of hostile assets given the specific 
threat. Once the patrol boxes were completed, the student assigned patrol craft to the 
assigned zones. This step was repeated for a second patrol box and a second patrol craft. 

During the generation of the scenario the SME observed the student to see if he was 
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completing the specified tasks as specified on SavageStudio Exercise Student Task 
Checklist. The SME also proceeded to ask the student questions of how the placement of 
his assets addresses, port vulnerabilities, critical asset protection, and establishment of the 
assessment, warning and threat zones. Once the student generated a scenario using 
SavageStudio, the SME proceeded to ask the student to navigate the 3D scene by using 
the viewpoint button. The student then observed the execution of the scenario. The SME 
provided feedback during scenario execution regarding the student placement of his 
assets in the scenario. The student launched the analyst report and analyzed scenario 
parameters and statistical results (boat interceptions, enemy success rate) to determine if 
the use of assets was optimized. The student was then asked to develop friendly Courses 
of Action (COAs) based on statistical results from the simulation and the student was 
allowed to go back to the scenario generation screen to implement the generated COAs. 
Each student then proceeded to observe the execution of the scenario and analyzed the 
results. After the scenario portion of the study was completed, the student was asked to 
complete a post-test. 

D. SUMMARY 

This chapter provided an in-depth look at the two studies that were conducted 
during this thesis research. Detailed information is provided about each study, namely 
Study IAT/EP Computer-based training versus Classroom instruction, and Study II 
Training effectiveness of the ATEP CBT with Simulation. Pertinent information is 
presented in detail for each study including participants, variables measured, apparatus 
and procedures used in the implementation of the lessons. 
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V. SCENARIO CREATION 


A. INTRODUCTION 

This chapter provides an in-depth step-by-step set of instructions for the creation 
of a scenario using SavageStudio. These steps include launching SavageStudio, creating a 
scenario, adding friendly and enemy forces, scenario execution, viewing the scenario, and 
loading a previously configured scenario from disk. 

B. LAUNCHING SAVAGESTUDIO 

The first step for scenario creation is to launch the tool. To launch the 
SavageStudio application the user clicks on the SavageStudio icon and launches the 
application. Upon launching the application a command window with scrolling text 
appears on the computer screen with the information that the application is being 
initialized. Once SavageStudio is initialized the main application content display window 
is displayed in the computer screen as shown in Figure 5. 



Figure 5. SavageStudio scenario generation display window presents a blank palette for 

scenario configuration. 
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C. SCENARIO CREATION 

For this scenario example the user creates a scenario located in Bremerton 
Washington, then adds the following entities: one Nimitz class Aircraft Carrier that acts 
as our High Value Unit (HVU), a patrol area with assigned specific dimensions, two 
patrol boats for perimeter defense assigned to patrol inside the patrol area, and two 
pleasure boats that are used as explosive-laden vessels by the hostile terrorist group. 

I. Select Location 

The first step in scenario creation is to select a port from the available listing of 
locations in the location menu selection list. Note each location may contain more than 
location or port. Instructions to follow. 

• First place the arrow pointer over the locations folder to select a location. 

• Select the locations folder by clicking the left mouse button over the key 
icon next to the locations folder as shown on Figure 6. All the locations 
contained in the folder appear on the screen below the Locations folder, 
as shown on Figure 7. 

• Select a specific location by clicking the left mouse button on the key 
icon next to location, select Bremerton. A red triangle followed by the 
word Bremerton appears on the computer screen. 

• By clicking the left mouse button over the key icon located next to the 
word Bremerton the map for Bremerton appears on your screen as shown 
on Figure 8. 
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Figure 6. SavageStudio scenario generation display window with the location menu 

selection folder opened. 
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Figure 7. SavageStudio scenario generation display, with left window depicting location 

and port menu-selection folders. 
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Figure 8. SavageStudio scenario generation display, depicting Bremerton port selection 
in the left hand pane and 2D display of Bremerton in the right hand pane. 

2. Add Friendly Forces 

The next step for scenario creation is to select a High Value Unit and to add other 
desired friendly forces from the available listing of robots, ships, tools, communications, 
and sensors in the models menu selection list. Instructions to follow. 

• First place the arrow pointer over the ship’s military folder to select a high 
value unit. 

• Select the Ship’s Military folder by clicking the left mouse button over the 
key icon next to the Ship’s Military folder as shown on Figure 9. All the 
ship models contained in the folder appear on the screen below the Ship’s 
Military folder. 

• Select a specific HVU by clicking the left mouse button on the key icon 
next to Carrier Nimitz United States. A white paper page icon followed by 
the word Nimitz CVN appears on the computer screen. 
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• By clicking the left mouse button over the key icon located next to the 
word Nimitz CVN the ship model for a Nimitz class Aircraft Carrier is 
displayed in the computer screen as shown on Figure 10. 

• The model for the Nimitz CVN is attached to the mouse pointer so the 
model moves as the mouse is moved over the application screen. 

• To disengage or deselect the model from the mouse pointer, press the Esc 
key on your keyboard by disengaging the model from the mouse pointer 
model movement ceases and the mouse is free to be used for other 
purposes on the application screen. 

• To engage or select the model again place the arrow pointer over the 
model and click the left mouse button, by completing this step the user is 
again allowed to move the model and place it anywhere in the application 
screen where the map is available. 
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Figure 9. SavageStudio scenario generation display window depicting the model, 
robots, ships, tools, communications and sensors menu folders and sub-folders. 
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Figure 10. SavageStudio scenario generation display window depicting a Nimitz class 
Aircraft Carrier selection in the left hand pane and 2D display of a Nimitz class 
Aircraft Carrier in the right hand pane. 

• To manipulate the models heading meaning to rotate the model, press the 
up/down arrow keys located in the keyboard. 

• Now that a HVU is selected the user will place the model next to a pier, 
note that the heading of the HVU is not the same as the pier located in the 
scenario. 

• Using the down arrow located in the keyboard change the model’s heading 

to match the pier’s heading, in this case the Nimitz CVN bow points south 
towards the water as shown on Figure 12. 
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Figure 11. SavageStudio scenario generation display window depicting a Nimitz class 

Aircraft Carrier rotating next to a pier. 
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Figure 12. SavageStudio scenario generation display window depicting a Nimitz class 
Aircraft Carrier located next to a pier selection in the left hand pane and the 
exposed metadata for the Nimitz class Aircraft Carrier in the right hand pane. 


Note to the right of the map screen the model’s information (SMAL metadata) is 
displayed while the model is selected. This information can be modified to assign a 
behavior or role to the particular model, so the next step is to assign the HVU behavior to 
this model. The default value for a model behavior parameter is set to patrol craft. 

• Place the arrow pointer in the model information screen; scroll down until 
the behavior parameter set is viewed in the computer screen. 
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• Under the Behavior parameter set there is a select box with the word type 
located next to it, select the folder icon. 

• Once the folder icon is selected a dialog box is displayed on the computer 
screen, it has a set of behavior libraries, select the Savage Defense tactics 
folder icon. 

• Once the folder icon is selected a dialog box is displayed on the computer 
screen, this folder has a set of tactics, select the friend folder icon. 

• Once the folder icon is selected a dialog box is displayed on the computer 
screen, the folder has a set of individual behavior tactics, select the 
HVU.xml icon. 

• Once the behavior is selected click on the save button, the dialog box will 
close and the application is back to the model information screen. 



Figure 13. SavageStudio scenario generation display window depicting the behavior 
libraries selection box that is being assigned to the Nimitz Aircraft Carrier. 
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Figure 14. SavageStudio scenario generation display window depicting the behavior 
libraries selection box and the friendly assets library selection sub-menu. 

The next step is to select patrol zones. For this example one patrol zone will be 
created. In this patrol zone the user places the military assets that patrol the area in order 
to protect the HVU. Patrol zones are used to determine defensive set up and measures. 
HVU Patrol zones are placed in the location where our assets will be more effective 
against an explosive laden vessel threat. 

• Place the arrow pointer over the patrol zone icon. 

• Select a patrol box by clicking the left mouse button over the key icon 
next to the patrol zone folder as shown on Figure 15. Select a patrol box 
by clicking the left mouse button on the white paper page icon followed 
by the word box, a red patrol box appears on your computer screen. 

• The model for the patrol box is attached to your mouse pointer so the 
model moves as you move the mouse pointer over the application screen. 

• To disengage or deselect the model from the mouse pointer, press the Esc 
key on your keyboard by disengaging the model from the mouse pointer 
model movement ceases and the mouse is free to be used for other 
purposes on the application screen. 
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• To engage or select the model again place the arrow pointer over the 
model and click the left mouse button, by completing this step the user is 
again allowed to move the model and place it anywhere in the application 
screen where the map is available. 

Note to the right of the map screen the patrol zone information (SMAL metadata) 
is displayed while the patrol box is selected. This information can be modified to assign 
the patrol zone different dimensions. The default value for a patrol zone is set to 50 
meters in the X direction, 0.01 meters in the Y direction and 50 meters in the Z direction. 

• For this example the user makes the patrol zone 250 meters in the X 
direction and 250 meters on the Z direction. 

• Place the arrow pointer in the model information screen select the X 
dimension box by clicking on top of it. Input the number 250 inside the 
box then press enter, repeat the same steps for the Z dimension, after 
completion of this steps the computer screen display looks similar to 
Figure 15. 



Figure 15. SavageStudio scenario generation display window with a patrol box added. 
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The next step is to select patrol assets that will be assigned to patrol the area 
inside the patrol zone. Available assets are contained in the model folders. For this 
example the user uses two patrol boats. The following steps are similar to the steps used 
to place the HVU and the patrol zone in the application screen, therefore fewer figures 
are provided. 

• Place the mouse arrow pointer over the ship’s military folder to select a 
patrol asset. 

• Select the ship’s military folder by clicking the left mouse button over the 
key icon next to the ship’s military folder. All the ship models contained 
in the folder appear on the computer screen below the ship’s military 
folder. 

• Select a specific patrol asset by clicking the left mouse button on the key 
icon next to Pearl Harbor security boat. A white paper page icon followed 
by the word Pearl Harbor security boat appears on your computer screen. 

• By clicking the left mouse button over the key icon located next to the 
word Pearl Harbor security boat the ship model for a security boat is 
displayed in your computer screen. 

• The model for the Pearl Harbor security boat is attached to your mouse 
pointer so the model will move as the mouse pointer is moved over the 
application screen. 

• To disengage or deselect the model from the mouse pointer, press the Esc 
key on your keyboard by disengaging the model from the mouse pointer 
model movement ceases and the mouse is free to be used for other 
purposes on the application screen. 

• To engage or select the model again place the arrow pointer over the 
model and click the left mouse button, by completing this step the user is 
again allowed to move the model and place it anywhere in the application 
screen where the map is available. 
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• To place the first security boat in the application screen, place the model 
on the location in the map. 

• Click the left mouse button; note the security boat is still attached to the 
mouse pointer to place the second security boat on the location map click 
the left mouse button as shown in Figure 16. 

• Press the Esc key to deselect the model. To manipulate the models 
heading as previously discussed press the up/down arrow keys located in 
the computer keyboard. 






n C£ 



Figure 16. SavageStudio scenario-generation display window depicting a Nimitz class 
Aircraft Carrier and the two added patrol boats on the left hand pane and the 
selected vehicle’s metadata on the right hand pane. 


Note to the right of the map screen the selected model’s information (SMAL 
metadata) is displayed. As previously discussed this information can be modified to 
assign a behavior or role to the particular model. The default value for a model behavior 
parameter type is set to patrol craft so for the two patrol boats the behavior parameter 
type does not need to be changed. 
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The patrol boats are assigned to patrol in the previously created patrol 


zone. 

• Under the behavior parameters for the patrol boat to the right of the 
computer screen, there is a white icon with a blue cross icon labeled 
Zones. 

• Click once on the icon and the ProbabilityZoneGeometry parameter is 
made available for modification and assignment, four parameters are 
displayed for the Zone. Next to entity ID a number is displayed, the 
number displayed is used to identify the asset in the simulation. 

• Place the mouse arrow pointer over the white square with red and blue 
squares connected by two green diamonds, click once on this square, the 
arrow pointer turns into the entity ID icon that was recently selected. 

• Place the entitylD icon on top of the patrol zone left click the mouse 
button and place (drop) the entitylD icon in the patrol zone located in the 
location map. 

3. Add Enemy Forces 

The next step for scenario creation is to select and add enemy forces from the 
available listing of robots, ships, tools, communications and sensors in the models menu 
selection list. Instructions to follow. 

• First place the arrow pointer over the ship’s civilian folder to select an 
enemy asset. 

• Select the ship’s civilian folder by clicking the left mouse button over the 
key icon next to the ship’s civilian folder as shown on Figure 14. All the 
ship models contained in the folder appear on the computer screen below 
the ship’s civilian folder. 
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DaDDDc: 


• Select a specific model by clicking the left mouse button on the key icon 
next to the speed boat. A white paper page icon followed by the word 
speed boat with driver high-definition appears on the computer screen. 


• By clicking the left mouse button over the key icon located next to the 
word speed boat with driver high-definition the ship model for a speed 
boat with a driver is displayed in the computer screen as shown on Figure 
17. 





Figure 


[j * 



17. SavageStudio scenario generation display window depicting a Nimitz class 
Aircraft Carrier, two added patrol boats and two speed boats on the left hand pane 
and the selected vehicle’s metadata on the right hand pane. 


• The model for the speed boat is attached to your mouse so the model 
moves as the mouse pointer is moved over the application screen. 

• To disengage or deselect the model from the mouse pointer, press the Esc 
key on your keyboard by disengaging the model from the mouse pointer 
model movement ceases and the mouse is free to be used for other 
purposes on the application screen. 
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• To engage or select the model again place the arrow pointer over the 
model and click the left mouse button, by completing this step the user is 
again allowed to move the model and place it anywhere in the application 
screen where the map is available. 

• To place the first speed boat in the application screen, place the model on 
the location in the map. 

• Click the left mouse button; note the speed boat is still attached to the 
mouse pointer to place the second speed boat on the location map click the 
left mouse button as shown in Figure 17. 

• Press the Esc key to deselect the model. 

• To manipulate the models heading as previously discussed press the 
up/down arrow keys located in the computer keyboard. 

Note the default behavior parameter for all surface ships is military patrol craft; 
therefore the behavior needs to be changed for the two enemy speed boats. The 
explosive-laden behavior is assigned to the two speed boats. To the right of the map 
screen the model’s information (SMAL metadata) is displayed while the model is 
selected. As previously discussed in the HVU model example this information can be 
modified to assign a behavior or role to the particular model, and so the next step is to 
assign the explosive-laden vessel behavior to this model. 

• Place the arrow pointer in the model information screen; scroll down until 
the behavior parameter set is viewed in the screen. 

• Under the Behavior parameter set there is a select box with the word type 
next to it, select the folder icon. Once the folder icon is selected a dialog 
box is displayed on the computer screen, it has a set of behavior libraries, 
select Savage tactics folder icon. 
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Figure 18. SavageStudio scenario-generation display window depicting a Nimitz class 
Aircraft Carrier, two added patrol boats and two speed boats and the behavior 
libraries selection box that is being assigned to the speed boats. 


Once the folder icon is selected a dialog is displayed on the computer screen, it 
has a set of tactics, select the hostile folder icon. 



Figure 19. SavageStudio scenario-generation display window depicting a Nimitz class 
Aircraft Carrier, two added patrol boats, two speed boats, the behavior libraries 
selection box, and Savage Tactics behavior sub-folder that contains the behavior 
that is being assigned to the speed boats. 
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Figure 20. SavageStudio scenario-generation display window depicting a Nimitz class 
Aircraft Carrier, two added patrol boats, two speed boats, the behavior libraries 
selection box. Savage Tactics behavior sub-folder and hostile behavior sub-folder 
that contains the behavior that is being assigned to the speed boats. 


• Once the folder icon is selected a dialog box is displayed on the computer 
screen, it has a set of individual behavior tactics, select the 
ExplosiveLadenVessel.xml icon. 

• Once the behavior is selected, click on the save button, the dialog box will 
close and the application will proceed to the model information screen. 

After completion of the previous steps, the location and the desired entities have 
been placed in the scenario. The entity behaviors have been modified based on the user 
preferences, and the scenario is configured to be run either visually in 3D or off-screen 
for statistics. 


57 

















4 Savat* 


n 0£ 

> 


D' 



O' 

5* 


ar.r 

a> 

ai 



Figure 21. 



SavageStudio scenario-generation display window depicting a fully 
configured scenario. 


4. Scenario-Execution Options 

The user has two scenario-executing options available depending on what the user 
wish to gain from the simulation. The first execution option available to the user is the 
3D view, which allows the user to accomplish specific tasks such as: visually inspect the 
entity behaviors and verify that the applied tactics are according to pre-planned 
responses, the user can view the simulation from the patrol boat point of view to 
experience what the patrol boat driver is receiving by visual sensors while the attack is 
taking place, the user can view the simulation from the explosive-laden vessel point of 
view to experience what the speed boat driver is receiving by visual sensors while the 
attack is being conducted. The user can view the scenario from any entity point of view 
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or from a top view (2D geographic view). In this way the user can gain visual insight to 
shortcomings on their defensive plans, or gain insight on the possible shortcomings on 
the terrorist attack plan. 

The second execution option available to the user is the statistical view. This 
option allows the user that is no longer interested in visual feedback to run the scenario 
faster and to analyze only the statistical results. The user can iterate over thousands of 
simulations in a matter of seconds. Some of the tasks that can be accomplished from this 
statistical view are: modifying a defensive tactic and analyze what is the percentage of 
success (stopping the enemy given the new tactic employment), given a configured 
scenario execute the scenario one thousand times and see the percentage that the enemy 
was able to destroy the HVU (enemy success percentage), the user places different assets 
to protect the HVU and analyzes which asset is better given the current scenario 
configurations. 

• To execute the scenario in a 3D view click on the green triangle inside the 
red circle icon located under the word help. 

• A Save As dialog box appears on the computer screen, name and save the 
configured scenario as shown in Figure 22. 



Figure 22. SavageStudio scenario-generation display window depicting a fully 
configured scenario that is being saved to the scenario libraries. 
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• Once the scenario is saved, the application launches Viskit assembly 
runner and the simulation viewer. 

• On the Viskit assembly runner screen, click on the gray box with a blue 
triangle inside it to start the simulation as shown on Figure 23. 

• To execute the scenario for a statistical view, click on the green triangle 
inside the matrix icon located next to the 3D view icon. 

• A save as dialog box appears on the computer screen, name and save the 
configured scenario as shown in Figure 23. 

• Once the scenario is saved the application launches Viskit assembly 
runner. 

• On the Viskit assembly runner screen, click on the gray box with a blue 
triangle inside it to start the simulation as shown on Figure 24. 

• SavageStudio Viskit assembly runner display window is displayed with 
the simulation results on the right hand pane of the screen inside the Viskit 
assembly runner screen. 



Figure 23. SavageStudio simulation visual display window depicting a fully configured 
scenario in the left-hand window and the Viskit Assembly Runner in separate 

right-hand window. 


60 



Vitkit Assembly Runr>»r: atfp.: 




Hi» edt u«^ 

Lv<^M Or«ipli fl<itor ' Lcitor Aswniiblv Aik#y%l Report 

LOCJlPurv SI DeSTi cf Evpaimenb ffiLJLTKtiClLRtierScto 


Sm itart orrw: 39 S 2992 

9m stop Omc 1000.0 
# r<plc-aoorn: 2 

VfrtMMOUlpot 
Sav« r^pAcAOon daita 
Pmt repAcjOcn report 
Pmt skjmmary reports 

■ ► 



NAVAl 

rOSTClRADUATI 

SCMCX 51 


tiue 


«trp.xxjcxxxxxxx3m «x««at>ly. 1 
4to»tonreitatU> • 0 
tMika»*taepllc«c.ieck« ■ t 

piincJi«pilc«tkofiD*pott» ' 

PKtnCaUMMKyf>*pOK« • ttvm 
•*ir«a«pllCBt.to«U>«t« • tevw 
• tlli0i«Sc«p « ^•l** 
aco^TlM • 1000.0 

v«rbo«« • (•!»• 

«uee*»*euliAtt*clL».»*«n ITALLY) 

J 1.0000 2.0000 l.»000 O.SOOO O.YOYl 

AUCC«a*CwlInt«ic«yc«.M«n (TAtLY| 

2 0.0000 0.0000 0.0000 0.0000 0.0000 

tapot C^dCoixtAct^.Man ITALL Y1 

2 0»0000 liOOOO O.SOOO 0.3000 O.TOTl 




lav*. Lw.0-**<*'l^***4 • 

* • (>i>t 

•« f ImT ItoV*' t lli'J •«f ti 


L 


Figure 24. SavageStudio Viskit Assembly Runner display window depicting the 

execution of the fully configured scenario. 


5. Scenario Viewing Options 

When the user executes the scenario in 3D viewing mode, the user has the ability 
to view the running simulation from different points of view. The user can view the 
scenario from any entity point of view or from a top view (2D geographic view) as 
previously discussed. The user can navigate and explore the scene; the user can examine 
the terrain, buildings, piers, ships and all the features available in the simulated 3D world. 
This is accomplished by using different viewpoints accompanying each model and 
created by the application. 
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Figure 25. SavageStudio simulation viewing display window depicting a first-person 
view of the simulation from the explosive-laden vessel. 



Figure 26. SavageStudio simulation viewing display window depicting a fully configured 
scenario in the right hand window and first person view of the simulation from the 
explosive-laden vessel in the left hand window. 
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Figure 27. SavageStudio simulation viewing display window depicting detonation of the 

explosive laden vessel at the impact point. 

D. LOADING A PREVIOUSLY CONFIGURED SCENARIO 

Using SavageStudio, the user has the ability to both configure and save scenarios. 
These saved scenarios can be loaded at a later time for viewing, modification and analyst 
review. If the user wants to add another patrol boat to the previously configured scenario, 
the user does not need to start a new scenario from the beginning. The user loads a 
previously configured scenario and adds the new entities or new parameters that he wants 
to change to the new scenario; this feature saves the user time when minimal changes are 
required. 

• To load a scenario from disk, the user launches SavageStudio once the 
application screen is loaded to the desktop, select the yellow folder icon. 

• Once the folder icon is selected a dialog box opens on the computer 
screen, select the desired scenario and click on save. 

• If the file loads satisfactorily the scenario configuration screen is 
automatically loaded with the desired scenario, further inspection of 
parameters and addition or detection of entries is permitted. 

• If no changes are required then execute the simulation using the same 
steps as previously discussed. 
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Figure 28. SavageStudio scenario generation display window depicting scenario loading 

from the scenario run library. 
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Figure 29. SavageStudio scenario generation display window depicting a fully 

configured scenario loaded from the run library. 
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E. SUMMARY 


This chapter provided a step by step set of instructions for the creation of a 
scenario using SavageStudio. These steps included launching SavageStudio, creating a 
scenario, adding friendly and enemy forces, scenario execution, viewing the scenario and 
loading a previously configured scenario from disk. 
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VI. RESULTS AND DISCUSSION 


A. INTRODUCTION 

This chapter provides an in-depth look at the two studies. Study IAT/FP 
Computer-based training versus Classroom instruction, and Study II Training 
effectiveness of the ATFP CBT with Simulation results and discussion. The chapter is 
divided in the following manner: Study I results and discussion then Study II results and 
discussion. 

Descriptive statistics are used to compare and analyze the data obtained from both 
studies. Descriptive statistics describe the basic features of the data and this evaluation 
method enabled us to present the data in an unbiased way. This approach makes it 
possible to determine the likelihood that the differences within and between the 
participant groups were caused by the variables studied and not just by chance. All tests 
are graded on a scale 0-100. 

B. STUDY I RESULTS 

The analysis of the Pre-Test for both groups indicated that the Cl students had a 
higher mean score than the CBT students. The CBT mean score was 50.55 and the Cl 
mean score was 52.85. The analysis of the Post-Test for both groups indicated that the 
CBT students had a higher mean score than the Cl students. The CBT mean score was 
68.88 and the Cl mean score was 60.71, resulting in net improvements of 18.33 and 7.86, 
respectively. 
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Figure 30. Pre-test and post-test mean scores for the CBT and Cl groups in study 1. 


The difference between the Pre-test and Post-test scores for the Cl groups reveal 
that there is increase in performance between subjects after the training session was 
conducted. The difference between the Pre-test and Post-test scores for the CBT groups 
reveal that there is increase in performance between subjects after the training session 
was conducted. 

The researcher conducted a two sample T-Test assuming equal variance with an 
alpha = 0.05 and 14 degrees of freedom. The T critical value found is Tc = 2.144 and the 
To= 1.306 as shown on Table 5. 

The difference between the Post-test scores for both the CBT and Cl groups 
reveal improvement in performance after the treatment was conducted. However due to 
high variance in the data further statistical analysis revealed that there is no significant 
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difference in performance in regard to the delivery method after the training session. 
Therefore T critical value was not attained then we have to accept the Null hypothesis 
that Ml = M 2 . 



Figure 31. CBT participant’s exam scores for study I. 



Figure 32. Cl participant’s exam scores for study I. 
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Histogram 



□ Frequency 


Figure 33. Frequency distribution of student’s pre/post-test scores difference for CBT 

participants. 


Histogram 



□ Frequency 


Figure 34. 


Frequency distribution of student’s pre/post-test scores difference for Cl 

participants. 
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CBT 

Classroom 

Mean 

18.33 

7.85 

Variance 

393.75 

65.47 

Observations 

9 

7 

Pooied Variance 

253.06 


Hypothesized Mean Difference 

0 


Df 

14 


t Stat 

1.30 


P(T<=t) one-taii 

0.10 


t Criticai one-taii 

1.76 


P(T<=t) two-taii 

0.21 


t Critical two-tail 

2.14 



Table 5. Two Sample T-Test results equal variance table for study 1. 


C. STUDY I DISCUSSION 

The results show statistically significant improvement of the participant’s 
performance on the tests after training for both CBT and CL Despite the apparent higher 
scores improvement of students receiving the CBT instruction there was no statistically 
significant difference between the increases in performance based on the delivery 
method. Differences in performance improvement based on the delivery method, can be 
argued that for this research none is found due to the small sample of participants or 
variability of the participant’s in the group. Nevertheless the results from the research 
given the circumstances mentioned and the literature research support the hypothesis that 
when the material presented is the same, CBT is as good as Cl. The practical application 
is that results from this study support the continued present use and expanded future use 
of CBT by the U.S. Navy. 

In many cases CBT is the best media because it allows easier access to the 
material. If it is on the Internet or on CD’s at the worksite, then training can be done on a 
flexible, short notice basis, without need for lengthy scheduling. In addition to being 

more flexible, CBT requires less time, budget and facilities. Time savings include; 
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eliminating travel time, the ability to start and stop training and the ability to skip 
material already mastered. Budget savings inelude; eosts to maintain elassroom faeilities, 
instruetor’s salaries, and students travel, lodging and lost work eosts. Based on these 
advantages and inereasing eonstraints on time and money, the implieation is that similar 
training for the U.S. Navy may transition to CBT in the future. 

D. STUDY II RESULTS 

The analysis of the pre-test and post-test seore data for the group that reeeived the 
AT/FP with simulation indicated that the post-test data scores are higher than pre-test 
data scores. The pre-test mean score was 41% and the post-test mean score was 84% a 
substantial increase in performance. The difference between the means after treatment is 
43% illustrated in Figure 35. 
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Figure 37. Frequency distribution of student’s pre/post-test scores difference for study IF 
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The difference between the pre-test and post-test scores for the group reveal that 
there is increase in performance between subjects after the training session was 
conducted. The difference between the pre-test and post-test scores can be attributed to 
the lesson provided for the students in the group. 

A T-Test paired two sample for means test assuming equal variance was 
conducted with an alpha = 0.05 and 19 degrees of freedom. The T critical value is found 
to be Tc = 2.09 and the To= 2.07E-10 (effectively zero) as shown in Table 6. 

The difference between the pre-test and post-test scores for the group reveal that 
there is a significant difference in performance after the lesson was administered. 
Therefore the Null hypothesis that Mi = M 2 is rejected and the alternative hypothesis is 
accepted. 



Variable 1 

Variable 2 

Mean 

6.15 

12.6 

Variance 

7.29 

2.25 

Observations 

20 

20 

Pearson Correiation 

0.483 


Hypothesized Mean Difference 

0 


df 

19 


tStat 

-12.15 


P(T<=t) one-taii 

1.04E-10 


t Criticai one-taii 

1.72 


P(T<=t) two-taii 

2.07E-10 


t Criticai two-taii 

2.09 



Table 6. T-Test paired two sample for student’s mean test result for study II. 


The tasks completion for the scenario portion of the study measuring skill 
objectives using SavageStudio reveals that 20 of 20 participants completed the objectives 
with a score of 80% or better. 3 of the 20 participants achieved a score of 83% and the 
other 17 participants score 100% completing all assigned tasks. The scenario task 
completion mean score is 97.5%. 
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Tasks 



asseesment 









zone 

warning zone 

threat zone 

ID oritical 


Observe the soenario 



subject 

established 

established 

established 

assets 

Id vulnerabilites 

and explain results 

score 

% Ber Student 

1 

1 

1 

1 

1 1 

1 

6 

1 

2 

1 

1 

1 

1 1 

1 

6 

1 

3 

1 

1 

1 

1 1 

0 

5 

0.83 

4 

1 

1 

1 

1 1 

0 

5 

0.83 

5 

1 

1 

1 

1 1 

1 

6 

1 

6 

1 

1 

1 

1 1 

1 

6 

1 

7 

1 

1 

1 

1 1 

1 

6 

1 

8 

1 

1 

1 

1 1 

1 

6 

1 

9 

1 

1 

1 

1 1 

1 

6 

1 

10 

1 

1 

1 

1 1 

1 

6 

1 

11 

1 

1 

1 

1 1 

1 

6 

1 

12 

1 

1 

1 

1 1 

1 

6 

1 

13 

1 

1 

1 

1 1 

1 

6 

1 

14 

1 

1 

1 

1 1 

1 

6 

1 

15 

1 

1 

1 

1 0 

1 

5 

0.83 

16 

1 

1 

1 

1 1 

1 

6 

1 

17 

1 

1 

1 

1 1 

1 

6 

1 

18 

1 

1 

1 

1 1 

1 

6 

1 

19 

1 

1 

1 

1 1 

1 

6 

1 

20 

1 

1 

1 

1 1 

1 

6 

1 

Score per Objective 

20 

20 

20 

20 

19 

18 



So per Objective 

1 

1 

1 

1 0.95 

0.9 


0.975 


Table 7. Student’s objective completion results using SavageStudio in Study II 


E. STUDY II DISCUSSION 

The results show statistically significant improvement of the participant’s 
performance on the specified post-tests after training was conducted. The performance 
improvement can be attributed to the lesson and the scenarios completed between tests. 
Results from the study and previous research on the subject support the findings that 
scores on post-test will be different than scores on the pre-test, if a treatment is provided 
to the group in this case the AT/FP lesson and scenario. The practical application for this 
study is that this research can be used to support future research where CBT and 
simulation effectiveness can be compared with Cl effectiveness. 

If future research finds that CBT and simulation is as effective as classroom 
instruction then future research ought to be conducted on the effectiveness and 
deployment of such systems for future Navy training. 

F. SUMMARY 

This chapter provides an in-depth look at the two studies, study I (AT/FP 
Computer-based training versus Classroom instruction) and Study II (Training 
effectiveness of the ATFP CBT with Simulation) results and discussion. Results for study 
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I showed that there is no significant difference between CBT and Cl if the content is the 
same. Results for study II showed that there is a difference in post-test scores after 
treatment is applied to a group (AT/FP lesson and simulation). 


76 



VII. CONCLUSION AND RECOMMENDATIONS 


A. CONCLUSIONS 

This thesis was motivated by the needs of the Navy in a period where Anti- 
Terrorism and Force Protection Training is a must, time and resources are scarce, and the 
Navy’s number-one priority is to train its sailors in an efficient manner. The enemies of 
the U.S. recognize that terrorism will continue to create problems for U.S. Forces 
worldwide, and the use of terrorist tactics will continue to be used in the future by these 
groups. 

The result of this research can help future development of CBT and simulation 
training systems by demonstrating a formal methodology to accomplish organizational 
requirements. The research conducted provided an in-depth look at the analysis, design, 
development, implementation and evaluation of the developed lessons for the AT/FP 
course using the ISD framework. The use of the ISD in the creation process of a CBT is 
imperative if successful, meaningful instruction is required. The CBT developed for this 
thesis research incorporated multimedia and simulation to enhance the knowledge gain 
and knowledge transfer. Future developers should remember Clark’s (1994) theory that 
well developed instruction is the most important aspect of training, more than the 
delivery method itself, and not let themselves fall in the trap to add more bells and 
whistles just for the sake of “coolness” if the content of the instruction is not well 
designed, then no matter what delivery method is used the instruction will not achieve its 
objectives. 

The results for the first study showed that there is no statistically significant 
difference between CBT and Cl if the content is the same. Results for study II showed 
that there is a difference in post-test scores after treatment is applied to a group (AT/FP 
lesson and simulation). This research study supports the theory that CBT is a viable 
alternative to Cl, and also supports the hypothesis that CBT and Simulation improves 
user learning and performance. These results support Clark’s (1994) theory, stated in the 
introduction, that media does not affect learning if the content of the instruction is the 
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same. Thus both this research and the literature, namely Greitier (2002) and Sitzmann et. 
al (2006) support the theory that performance can be effectively increased using either 
type of instruction delivery method. 

Results and research also support the Navy’s vision of the Training Revolution 
(Executive Review of Navy Training 2001). Leaders face budget and operational time 
constraints that force them to work in unconventional ways to accomplish the Navy’s 
mission. Computer-Based Training and Simulation is one of the unconventional 
approaches that offer several advantages for the military organization. Such type of 
training offers leaders savings on resources, time and money. The necessary supporting 
technological infrastructure is already in place on Navy ships and sore bases. The goal of 
CBT is not to eliminate Classroom Instruction altogether but, rather to supplement it or 
make available the instruction to the sailor when other methods are not available. 

The CBT developed for this thesis research incorporated multimedia and X3D 
graphics simulation to enhance the knowledge gain and knowledge transfer. 

B. RECOMENDATIONS FOR FUTURE WORK 

1. Human in the Loop for Training 

During the execution of this thesis research, agent and security postures during 
scenario creation and execution were not modifiable, and so the participant was only able 
to select behaviors or security postures from the Savage libraries. It is recommended that 
mission decision points be added to SavageStudio scenarios where the user has the ability 
to manipulate agent actions or modify security postures at critical decision points, then let 
the simulation execute and see the results of the user decisions. This feature allows the 
user to further exercise course of action and planning skills. During the execution of this 
thesis research the SME observed the participant’s actions during scenario creation and 
execution and listened to the participant’s verbal explanation of his actions. The SME 
collected the participant’s information by annotating the information in a notebook for 
later debrief and feedback on the user performance. It is recommended that feedback 
screens and replay options be provided to the instructor allowing him to go over the user 
decisions and use the features to provide feedback and teaching points to the user. During 
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the execution of this thesis research information was provided to the participant using 
paper created documents referenced in Appendix J thru Appendix M. It is recommended 
that information be provided to the user by creating a tab section during scenario creation 
that contains the following information: Commanding Officer’s Guidance tab, ATO 
Mission Objectives tab, Threat Assessment tab and Intelligence Briefs tab. 

These tabs can be updated as the simulation is executed, for example, new 
intelligence information is made available to the user, so that the user may change force 
protection posture based on the new intelligence. 

2. Expand the Behavior Libraries 

It is recommended that the SavageStudio be enhanced by authoring additional 
behavior libraries to include the following behaviors: 

1. Ship’s personnel responses against low slow flyer. 

2. Ship’s security force personnel responses against swimmer attack. 

3. Security boat responses against swimmer attack. 

4. Hostile swimmer attack behavior. 

3. Migrate SavageStudio Application for Shore Facilities 

It is recommended that the approach and framework used in SavageStudio be 
migrated to shore facilities for the protection of high value assets in land. The 
technologies used in the creation for SavageStudio are well suited for similar AT/FP 
applications for shore facilities. Representation of shore and land facilities can be created 
with technology being researched by the Savage Group at the Naval Postgraduate School. 
The behavior libraries can be easily modified and adapted to behaviors used in AT/FP of 
land facilities. 

4. Design of Experiments (DOE) 

In order to test specific variable effects when conducting risk and cost assessment 

there exists a need to keep all variable effects constant except for the tested variables. 

Currently some of the algorithms used in SavageStudio use stochastic processes which 
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use probabilistic methods to generate states in the simulation. While this is good for 
course of action and training applications, the additional ability to only modify one 
variable at a time is good for testing risk assessment. For example, if a friendly defense 
posture is maintained constant and the departure of the explosive laden vessel constant, 
the user can modify the location of security assets between runs, and then the analyst can 
determine if the modification of location of these assets benefited or hampered a given 
defense plan. 

5. Mission Playback 

In order to provide trainees with feedback there is a need to add a mission 
playback feature, so that either instructor or trainees can go back and review specific 
decision points during the simulation execution. 

6. Larger Sample Size 

It is recommended that this study be replicated with a larger sample size and true 
random assignment to treatment groups. Future research that builds on Clark’s theory 
testing varying levels of complexity of the material presented. There may be a point 
where CBT and Cl do not perform equally, and indeed a well-designed CBT might 
outperform Cl, as indicated in Study I. Maximizing CBT effectiveness deserves serious 
efforts since the potential payoffs are so significant. 
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APPENDIX A. TASK ANALYSIS FOR SAMPLE LESSON 2 OF 
THE ATO LEVEL II COURSE USING CBT AND X3D GRAPHICS 


Task Analysis: 

The following task analysis identifies the required tasks that the user will be expected to 
perform after completion of the lesson. 

Training Objectives: 

Course training objective 2.0: 

Develop and implement Pre-Planned Responses that will be effective mechanisms to 
detect, deter, defend and mitigate likely terrorist threats against U.S. Navy assets and 
installations. 

Course training objective 3.0: 

Assess the anti-terrorism environment, situation for an afloat command, given current 
contingencies and sources of information. 

The learning objectives that will be measured will be knowledge and skill based 
objectives. 

Learning objectives: 

2.1 Define the importance of the Defense in Depth concept in accordance with Navy 
Tactics, Techniques and Anti-terrorism/Force Protection procedures and apply 
this concept to ship in-port security. (Knowledge) Performance data for this 
objective will be collected in the post-test. 

2.2a Define the principles of the Defense in Depth concept in accordance with Navy 
Tactics, Techniques and Procedures Anti-terrorism/Force Protection and apply 
this concept to ship in-port security scenario. (Knowledge) Performance data for 
this objective will be collected in the post-test. 

2.2b Use the SavageStudio in visual mode to explain the principles of defense in depth. 
(Knowledge) Performance data for this objective will be collected using the 
SavageStudio, student will explain the defense in depth concept as it applies to the 
scenario. 

2.2c Use the SavageStudio in visual mode to assess and demonstrate the application of 
the principles of defense in depth as they occur in a scenario. (Skill) Performance 
data for this objective will be collected using the SavageStudio, student will 
assess the scenario situation and will demonstrate the application of the defense in 
depth principles by the correct placement of assets as it applies to the scenario. 
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2.3a Define the Assessment, Warning and Threat Zones as used in the anti-terrorism 

program. (Knowledge) Performance data for this objective will be collected in the 
post-test. 

2.3b Use the SavageStudio visual mode to demonstrate the assessment, warning and 
threat zones as used in the anti-terrorism program concept of defense in depth. 
(Knowledge) Performance data for this objective will be collected using the 
SavageStudio, student will explain or address the 3 zones as they apply to the 
scenario. 

2.3c Use the SavageStudio in visual mode to demonstrate the application of the 

Assessment, Warning and Threat Zones as used in the anti-terrorism program. 
(Skill) Performance data for this objective will be collected using the 
SavageStudio, student will demonstrate or identify the 3 zones as they apply to 
the scenario. 

2.4a Define the purpose of pre-planned responses in accordance with Navy Tactics, 
Techniques and Anti-terrorism/Force Protection and NTTP 3-07.2.1 (Rev A). 
(Knowledge). Performance data for this objective will be collected in the post¬ 
test. 

2.4b Define the likely terrorist threats by applying the pre-planned responses concept 
in accordance with Navy Tactics, Techniques and Procedures Anti¬ 
terrorism/Force Protection. (Knowledge) Performance data for this objective will 
be collected in the post-test. 

2.5a Use the SavageStudio in visual mode to identify critical assets, and vulnerabilities 
for the NAVSTA scenario. (Knowledge) Performance data for this objective will 
be collected using the SavageStudio, describe and explain the assessment of the 
environment and situation, the student will explain potential threats and 
vulnerabilities as they apply to the scenario. 

2.5b, Use the SavageStudio in visual mode to explain the anti-terrorism environment 
and situation; using the X3D visual scene of the generated scenario explain the 
force protection employment of assets to protect given critical assets while 
minimizing vulnerabilities. (Skill) Performance data for this objective will be 
collected using the SavageStudio, the student will identify potential threats and 
vulnerabilities as they apply to the scenario. 
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Task 1: 

At the end of task 1 the user will be expected to: 

1. Complete the pre-test 

Task 1. Complete the Pre-Test: 

1.1 Complete 20 question test. 

1.2 Inform the instructor of pre-test completion. 


Task 2: 

At the end of task 2 the user will be expected to: 

1. Be able to answer questions and state the purpose of Defense in Depth. 

2. Be able to answer questions and describe the principles of Defense in Depth. 

3. Be able to answer questions and describe the Assessment, Warning and Threat Zones. 

4. Be able to answer questions and describe the purpose of pre-planned responses. 

5. Be able to answer questions and describe the likely terrorist threats for developing 

pre-planned responses. 

Task 2. Complete the CBT portion of the lesson: 

2.1 Load the CBT lesson in the computer. 

2.2 Launch the AT/FP CBT course using the mouse. 

2.3 Select lesson 2 from the CBT menu. 

2.3 Start the lesson. 

2.4 Observe and listen to the lesson. 

2.5 Once the lesson is completed, contact the SME for further instructions. 

Task 3: 

At the end of task 3 the user will be expected to: 

1. Use SavageStudio in visual mode to assess the anti-terrorism environment and 
situation, using visual scene of port maps to identify critical assets and vulnerabilities. 

2. Use SavageStudio in visual mode to describe the principles of defense in depth as they 
occur in a scenario. 

3. Use SavageStudio in visual mode to describe the assessment, warning and threat zones. 

4. Use SavageStudio in visual mode to exercise pre-planned responses. 

5. Set up defense in depth for the NAVSTA scenario and list and observe the pre-planned 
responses for boat-bom lED. 

Task 3. Use SavageStudio to generate and execute a scenario: 

3.1 Read the scenario information given by the instructor. 

3.2 Identify critical assets and vulnerabilities. 

3.3 Prioritize identified critical assets. 

3.4 Select location to place critical assets for maximum security. 

3.5 Set up defense in depth using patrol boxes, patrol boats, radars and barriers. 

3.5.1 Place the patrol box within the chosen harbor. 

3.5.1.1 Assign the friendly asset to the patrol zone. 

3.5.2 Modify the patrol zone dimensions. 


83 



3.5.3 Place barriers within the chosen harbor. 

3.5.4 Place radar assets within the chosen harbor. 

3.5.5 Repeat task 3.5.1 for a second friendly asset (patrol boat). 

3.6 Execute the scenario using the visual mode. 

3.7 Navigate the scene using the viewpoint button. 

3.7.1 Observe patrol boat pre-planned response. 

3.7.1.1 Observe patrol boat intercept unauthorized vessels. 

3.7.1.2 Observe patrol boat shoulder unauthorized vessels. 

3.7.1.3 Observe patrol boat or SSDF engage unauthorized vessel 
with destructive fire. 

3.8 Launch and review the analyst report. 

3.9 Analyze threat and simulation parameters using the analyst report. 

3.9.1 Determine patrol boat interceptions. 

3.9.2 Determine enemy success percentage. 

3.9.3 Determine if use of assets was optimized. 

3.10 Develop friendly courses of action based on statistical results from the 
simulation. 


Task 4: 

At the end of task 4 the user will be expected to: 

1. Complete the post-test 

Task 4. Complete the Post-Test: 

4.1 Complete 20 question test. 

4.2 Inform the instructor of post-test completion. 
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APPENDIX B. LESSON-ENABLING OBJECTIVES 


LESSON-ENABLING OBJECTIVES 

1. Enabling Objectives that will Accomplish the Second Learning 

Objective 

1. Use the SavageStudio visual mode to describe the principles of defense 
in depth as they occur in a scenario. 

2. Use the SavageStudio visual mode to describe the assessment, warning 
and threat zones as used in the anti-terrorism program concept of defense in depth in 
accordance with Navy Tactics. 

3. Use the SavageStudio visual mode to describe the Naval Vessel 
Protection Zone in accordance with Navy Tactics. 

4. Use the SavageStudio visual mode to describe the purpose of pre¬ 
planned responses in accordance with Navy Tactics. 

5. Use SavageStudio to analyze course of action (COA) development and 
implementation of Pre-Planned responses, identification of critical assets, identification 
and quantification of threat, assessing asset vulnerability, assessing risk and developing 
countermeasures. 

6. Use the SavageStudio visual mode to compare and select COA in 
accordance with Navy Tactics. 

2. Enabling Objectives that will Accomplish the Third Learning 

Objective: 

1. Use the SavageStudio to visual mode to assess the anti-terrorism 
environment and situation, using visual scene of port maps to identify critical assets and 
vulnerabilities. 

2. Set up defense in depth for the NAVSTA scenario and list and observe 
the pre-planned responses for boat-born lED. 
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3. Enabling Objectives that will Accomplish the Fourth Learning 
Objective: 

1. Use the SavageStudio visual mode to analyze COAs that will be used in 
the development of the AT/FP plan. 
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APPENDIX C. MEDIA SELECTION 


The methodology used in media selection is described below for knowledge- 
based objectives and skill-based objectives. The process was repeated for each objective 
to select the most effective media delivery method. 

A. KNOWLEDGE-BASED OBJECTIVE 

1. Define the objective 

Define the importance of defense in depth 

2. Classify the objective 

Facts about defense (declarative knowledge objective) 

3. Strategy 

List and explain important reasons 

4. Media Delivery 

Classroom lecture, CBT, textbook, audio visual 

5. Content 

List, narrative explanation, case examples 

6. Performance Measurement 

Score on written test, instructor evaluation 

B. SKILL-BASED OBJECTIVE 

1. Define the objective 

Assess and demonstrate the application of defense in depth as it occurs in a 
scenario 

2. Classify the objective 

Tactical concepts about defense in depth (skill objective) 
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3. Strategy 

List and explain operating procedures, show how to implement DID (process) 

Demonstrate correct threat ID, set defenses lAW with threat assessment, 
operate the SIM and execute plan, analyze results then correct tactical 
planning and execute changes 

4. Media Delivery 

Classroom lecture. Scenario and Simulation 

5. Content 

Narrative explanation, procedure case examples 

6. Performance Measurement 

SME evaluation 
System statistical tally 
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APPENDIX D. STORYBOARD FOR TRAINING AND 
CLASSROOM INSTRUCTION COMPARISON STUDY 



CENTER EOR SECURITY EORCES 


Introduction to Terrorism 
Lesson Topic 
1.0 


Presentation Details: 

Slides: 35 
Duration: 00:14:25 
Filename: 

MAR07\OA3402PerfAssesm\finalproject\Finalpilotstudyppt.ppt 

Presenter Details: 

Name: 


C:\Cruz\JAN- 
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Slide 1 ® 

Slide 1 

Duration: 00:00:25 

Advance mode: Auto 

CENTER EOR SECURITY EORCES 


Introduction to Terrorism 

Lesson Topic 

1.0 

1-2-1 


Terrorism is nothing new, its deadiy acts have been around for hundreds of years and acts of 
terrorism against Americans and other nations wiii continue for many years to come. 
Unfortunateiy for the innocent, terrorist of today are becoming more innovative in the carrying 
out of their deadiy acts. The end resuit are scenes of massive carnage. 


Slide 2 « 

Slide 2 

Duration: 00:00:05 

Advance mode: Auto 

References 

• DOD 0-2000.12-H 

• NWP 3-07.2. 

• 9-11 Commission Report 

• USATRADOC DCINST No. 1, 

Version 3.0 

1-2-2 


References for this iecture were taken from the following sources 


Slide 3 ^ 

Slide 3 

Duration: 00:00:10 

Advance mode: Auto 

Enabling Obiectives 

• DEFINE terrorism 

• IDENTIFY the categories of terrorist 
organizations 

• DESCRIBE types of terrorist training 

1-2-3 

No notes for this slide 
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Slide 4 9 
Slide 4 

Duration: 00:00:17 
Advance mode: Auto 



Enabling Objectives 


IDENTIFY the general characteristics 

of a terrorist operation 

LIST the seven phases of a terrorist 

operation 

DESCRIBE common terrorist attacks 
DESCRIBE the types of terrorist threats 
DESCRIBE the future trend of terrorism 


No notes for this slide 


Slide 5 « 

Slide 5 

Duration: 00:00:18 
Advance mode: Auto 



“The calculated use of unlawful violence or threat of unlawful 
violence to inculcate fear; intended to coerce or intimidate 
governments or societies in the pursuit of goals that are 
generally political, religious, or ideological.” 

1-2-S 


You wiil take command at a time in which events combine to make terrorism a top nationai 
security concern. Since the Beirut bombing in 1983, the U.S. miiitary has suffered more 
casuaities to terrorism than in combat operations. At a time when its miiitary strength is 
unmatched as the world’s remaining superpower, the United States is committed to more 
unfamiliar environments than at any time in its history due to widespread political instability 
around the world. Rogue state and sub-state opponents of U.S. policy cannot challenge U.S. 
military power directly, but they can and do attack vulnerabilities of U.S. Forces in an effort to 
erode military effectiveness and morale. 


Definition of Terrorism 

Terrorism has always been difficult to define. Even within the U.S. Government, different 
departments adhere to different definitions. 

The DoD defines terrorism as “The calculated use of violence or threat of violence to inculcate 
fear; intended to coerce or to intimidate governments or societies in the pursuit of goals that are 
generally political, religious or ideological”. 
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Slide 6 9 

Slide 6 

Duration: 00:00:36 

Advance mode: Auto 

Common Elements of Terrorism 

• Political 

• Psychological 

• Violent 

• Dynamic 

• Deliberate 

1-2-6 

Elements of Terrorism: 

The definition in the previous slide is clear and comprehensive. It contains the three elements that 
should be included in any definition of terrorism: 

1. Political - Clausewitz’ statement that “war is a continuation of policy by other means” is taken 
as a truism by terrorists. 

2. Psychological - The intended result is to create a psychological 

Terror 

3. Violent- Violence, coercion, and destruction are used in the commission of the act to produce 
the desired effect 

4. Dynamic - Terrorist groups demand change, revolution or political movement. 

5. Deliberate - Terrorism is an activity planned and intended to achieve particular goals. 

Slide 7 « 

Slide 7 

Duration: 00:00:10 

Advance mode: Auto 

Categories of Terrorist Organizations 

• Government affiliation categories 

• Motivation categories 

• Ideological categories 

1-2-7 

These topics will be discussed in the next few slides: 
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Slide 8 9 

Slide 8 

Duration: 00:00:30 

Advance mode: Auto 

Government Affiliation 

• Non-state supported 

• State supported 

• State directed 

1-2-8 

Non-state supported. These are terrorist groups that operate autonomousiy, receiving no 
significant support from any government. 

• State-supported. These are groups that generaiiy operate independentiy but receive 
support from one or more governments. 

• State-directed. These groups operate as an agent of a government and receive substantiai 
inteiiigence, iogistic, and operationai 

support from the sponsoring government. 

Slide 9 ® 

Slide 9 

Duration: 00:01:35 

Advance mode: Auto 

Motivation Categories 

• Separatist 

• Ethnocentric 

• Nationalistic 

• Revolutionary 

1-2-9 

Separatist. Separatist groups are those with the goal of separation from existing entities through 
independence, political autonomy, or religious freedom or domination. The ideologies separatists 
subscribe to include social justice or equity, anti-imperialism, as well as the resistance to 
conquest or occupation by a foreign power. 

• Ethnocentric. Groups of this persuasion see race as the defining characteristic of a society, and 
therefore a basis of cohesion. There is usually the attitude that a particular group is superior 
because of its inherent racial characteristics. Examples: KKK, Aryan Nation, Black Panthers, 

• Nationalistic. The loyalty and devotion to a nation, and the national consciousness derived from 
placing one nation’s culture and interests above those of other nations or groups is the motivating 
factor behind these groups. 

This can find expression in the creation of a new nation, or in splitting away part of an existing 
state to join with another that shares the perceived “national” identity. 

• Revolutionary: These groups are dedicated to the overthrow of an established order and 
replacing it with a new political or social structure. Although often associated with communist 
political ideologies, this is not always the case, and other political movements can advocate 
revolutionary methods to achieve their goals. 
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SlidelO 

Choices 

Interaction type: Choices 

Passing score: 60 

Instructions: None 

Custom Message: None 

7 7 

Learning Game Placeholder 
Learning Game: Choices 

Title: Choices 

L J 

Questions 

1. The foliowing elements are essential in the definition of terrorism in accordance to the 
Department of Defense: 

violence or threat of violence, fear, coercion, pursuit of goals 

1) True (Correct) 

2) False 

2. Which is a category of terrorist organization? 

1) government affiliated (Correct) 

2) state directed 

3) separatist 

4) anarchist 

3. Which is a characteristic of a government affiliated terrorist organization? 

1) state support (Correct) 

2) violence fear 

3) anarchist 

4) lED 

Slidell ® 

Slide 11 

Duration: 00:00:06 

Advance mode: Auto 

Ideological 

• Political 

• Religion 

• Social 

1-2-11 

Ideological categories describe the political, religious, or social orientation of the group. 
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Political 


Slide 12 « 
Slide 12 



Duration: 00:00:52 
Advance mode: Auto 



Right Wing 
Left Wing 
Anarchist 



Right wing: These groups are associated with the reactionary or conservative side of the poiiticai 
spectrum, and often, but not exciusiveiy, are associated with fascism or neo- Nazism. 

Left wing: These groups are usuaiiy associated with revoiutionary sociaiism or variants of 
communism. 

Anarchist: Anarchist groups are anti-authority or anti-government, and strongiy support individuai 
iiberty and voiuntary association of cooperative groups. _ 

Slide 13 « 

Slide 13 



Religious 


Duration: 00:00:13 
Advance mode: Auto 



Religiously inspired terrorism is on the rise, with a forty-three percent increase of total 
international terror groups espousing religious motivation between 1980 and 1995. While Islamic 
terrorists and organizations have been the most active, and the greatest recent threat to the 
United States, all of the major world religions have extremists that have taken up violence to 
further their perceived religious goals. Religiously motivated terrorists see their objectives as holy 
writ, and therefore infallible and non-negotiable. Religious motivations can also be tied to ethnic 
and nationalist identities, such as Kashmiri separatists combining their desire to break away from 
India with the religious conflict between Islam and Hinduism. The conflict in Northern Ireland also 
provides an example of the mingling of religious identity with nationalist motivations. There are 
frequently instances where groups with the same general goal, such as Kashmiri independence, 
will engage in conflict over the nature of that goal (religious or secular government).Christian, 
Jewish, Sikh, Hindu and a host of lesser known denominations have either seen activists commit 
terrorism in their name, or spawned cults professing adherence to the larger religion while 
following unique interpretations of that particular religion’s dogma. Cults that adopt terrorism are 
often apocalyptic in their worldview, and are highly dangerous and unpredictable. It is interesting 
to note that religiously motivated terrorists are among the most energetic developers of Weapons 
of Mass Destruction (WMD) for terrorist use. Also, religiously inspired cults executed the first 
confirmed uses of biological and chemical nerve agents by terrorists. _ 
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Slide 14 « 

Slide 14 

Duration: 00:00:17 

Advance mode: Auto 

Social 

W 

Earth Liberation Front 
(ELF) 

Animal Rights Groups 

Pro-Choice 

Often particular social policies or issues will be so contentious that they will incite extremist 
behavior and terrorism. Frequently this is referred to as “single issue” or “special interest” 
terrorism. Some issues that have produced terrorist activities in the United States and other 
countries are: 

• Animal rights 

• Abortion 

• Ecology/environment 

• Minority rights 

SlidelS 

Choices 

Interaction type: Choices 

Passing score: 80 

Instructions: None 

Custom Message: None 

I 

Learning Game Placeholder 
Learning Game: Choices 

Title: Choices 

1 

Questions 

1. Which is included in a ideological affiliated terrorist organization? 

1) religious (Correct) 

2) rationalist 

3) anarchist 

4) political 

2. Which is a political affiliated terrorist organization? 

1) anarchist 

2) ideological 

3) right wing (Correct) 

4) PETA 

3. These are included in a government affiliated terrorist organization except: 

1 ) separatist 

2) ethnocentric groups 

3) revolutionary 

4) political (Correct) 
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Slide 16^ 

Immediate Goals and Common 
Characteristics of Terrorist Groups 

Duration: 00:00:17 

Advance mode: Auto 

Immediate Goals and 
Common Characteristics 
of Terrorist Groups 

1-2-16 

Terrorist groups, like any other organization have both immediate and long-range goals. 

Therefore, each act of terrorism is designed to accomplish something specific as part of an 
overall strategy. These immediate or short-term goals usually include one or more of the 
following: (NEXT SLIDE) 

Slide 17 « 

Slide 17 

Duration: 00:01:52 

Advance mode: Auto 

Immediate Goals 

• Recognition 

• Eliminate western influence 

• Over-reaction of government 

• Harass, weaken, or embarrass government 

• Destroy facilities, disrupt communications 

• Discourage foreign investors 

• Influence government decisions 

• Free prisoners 

• Satisfy vengeance 

1-2-17 

Obtain worldwide or local recognition for the groups cause. The publicity advertises the 
movement and is of utmost importance to the group. Eliminate western influence in a region. 
Osama bin Laden’s goal is to get the United States and Israel out of the Middle East and 
establish a radical fundamental Islamic regime with out borders throughout the Middle East. 

Cause an over-reaction by the government: This is one of the basic precepts of revolutionary 
strategies. Over-reaction to terrorist activities may result in oppressive and restrictive population 
control measures, such as instituting special antiterrorist laws that allow searches warrants, 
detention of citizens without charges, curfews, and travel restrictions. However, people see these 
measures as a violation of their rights, and they also lend an element of legitimacy to grievances 
of the movement, creating an atmosphere more sympathetic to the terrorist cause. Harass, 
weaken or embarrass government security forces. Such tactics as calling in false alarms, 
assassinating security personnel, and conducting sophisticated raids and ambushes are used to 
reduce the effectiveness of security forces and erode public confidence in them Obtain money 
and equipment. Terrorist acts may be used to steal or extort money and equipment, especially 
weapons and ammunition essential to the struggle. It was paid in 1974 by the Exxon Oil 

Corporation for the release of Victor Samuelson, a company executive working in Argentina who 
had been held for 144 days by the Argentine People’s Revolutionary Army. Destroy facilities and 
disrupt lines of communication. This tactic obviously hinders the government’s ability to control 
the violence, both militarily and politically. It creates a chaotic atmosphere within the population in 
which entire Discourage foreign Investments or assistance programs. If this goal is achieved, the 
economy of the target country is affected, causing an increase in unemployment and a decrease 
in government revenues. Influence government decisions, legislation or elections. There is little 
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doubt the bombing of the Marine barracks in Beirut influenced our government’s decision to 
eventually withdraw the Marines __ 

Slide 18 % 

Slide 18 

Duration: 00:00:38 
Advance mode: Auto 


Characteristics of Terrorist Groups 
Terrorists seek to intimidate populations by promoting and spreading fear. They are militarily 
weaker. This means they must resort to hit-and-run guerrilla tactics and will be reluctant to take 
up a static defensive posture against the stronger enemy unless they possess a “bargaining chip” 
such as hostages. They are usually urban based. This offers access to modern means of 
transportation, money-laundering mechanisms, communications systems, and international 
contacts. Even rural-based insurgent movements will have urban-based terrorist cells that provide 
special support to the movement. They are highly mobile. Being urban-based, the terrorists have 
access to the transportation systems and the international contacts that allow them to obtain the 
documents they need to travel frequently to other cities and countries for operations or in search 
of safe havens and support. They usually act covertly. While the clandestine nature of the terrorist 
groups contributes to their flexibility and effectiveness, it can also be a problem. These groups 
are forced to maintain and support intricate operational systems complete with personnel and 
assets. _ 

Slide 19® 

Typical Training Programs 

Duration: 00:00:37 
Advance mode: Auto 


Typical training programs used by the terrorist groups at the organizational and individual levels 
Terrorist training programs being paramilitary in nature are oriented to the mission, the physical 
characteristics of the area of operations, and the security environment in the operational area. 
Most of the training is somewhat standardized. Historically, works such as the Mini-Manual of the 
Urban Guerrilla by Brazilian terrorist Carlos Marighella, and the writings of revolutionary leaders 
Mao Tse-tung, Che Guevera, and Fidel Castro are used as instructional texts and indoctrination 
material. Discipline is strictly enforced, and trainees are closely monitored for signs of weakness 
in their ability or ideological commitment. Internal training is conducted at the cell level by 
members of the group within the target country, usually in a clandestine fashion. 

External Training: Non-group members outside the operational area conduct external training. It 



• Internal 

• External 



Characteristics 


Seek to intimidate 
Militarily weaker 
Urban based 
Highly mobile 
Well trained 
Covert 


98 













can take two different forms. First, the operatives may travei abroad, individuaiiy or as a group, to 
training camps or to other permanent institutions for their training. 


Slide 20 ^ 
Typical Training 

Duration: 00:00:21 
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Typical Training 

Physical conditioning 
Weapons and explosive training 
Political indoctrination 
Combat tactics and techniques 
Psychological warfare 
Survival skills 
Communications 
Specialized training 


A typical individual terrorist training program includes most, if not all of the following: 

1. Physical conditioning. 

2. Weapons and explosive training, including how to purchase or manufacture explosives to 
construct improvised explosive devices. 

3. Political indoctrination. 

4. General tactics and combat techniques, including instruction on how authorities usually 
respond to hostage-barricade situations.5. Trade/crafts, including placing or recovering caches, 
opening mail covertly, surreptitious entry to homes or offices and wire-tapping. 

6. Psychological warfare. 

7. Survival skills, including survival in urban and foreign environments.8. Communications, 
including radio, telephone, “dead drops”, computers and satellite. F. Since terrorist operations are 
paramilitary in nature, it is not surprising that there are many similarities in the training of terrorist 
and in the training of Special Operations Forces of the U.S. In fact, terrorists are known to have 
capitalized on the open nature of our society and our willingness to publish potentially damaging 
material such as the Anarchist’s Cook Book, Poor Man’s James Bond and versions of the Special 
Forces and Ranger operational manuals. 
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Choices 

Interaction type: Choices 
Passing score: 80 
Instructions: None 
Custom Message: None 


Learning Game Placeholder 
Learning Game: Choices 
Title: Choices 


1. Which 2 training programs are not in a typical terrorist training program? 

1) Physical condition, survival skills 

2) Communications, political indoctrination 

3) Electronic, network warfare (Correct) 

4) Weapons, explosive training 

2. In accordance with the DoD the definition of external terrorist is: 

Non-group members outside the operational area conduct external training. It can take two 
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different forms. First, the operatives may travel abroad, individually or as a group, to tr 

1) True (Correct) 

2) False 
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Terrorist 
Operations 
and it’s 

Planning Cycle 


1-2-22 

Will be discussed in the next few slides 
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“The main point is to select 
targets where success is 100% 
assured.” 

Dr. George Habash, Founder. PFLP 
(Popular Front for the Liberation of Palestine} 






Terrorist Planning 
Cycle 


Brotd 
I TargM 
S ^k ct ion 


No notes for this slide 
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Broad 


1 Target 


Selection 


Duration: 00:00:35 
Advance mode: Auto 


Collection of information on a large number 
of potential targets 


• Also includes open source and general info 
collection 


- Media stories 


- Internet 


• Potential targets may be screened based on 

- Symbolic value 

- Potential for high media value 

- Objectives of group 


This phase is the collection of information on a 


arge number of potential targets, some of which 


may never be attacked, or seriously considered for attack. 

This phase also includes open source and general information collection. Some features of this 
type 

•Stories from newspapers, other media, and journalistic sources often provide key 
•Internet research provides texts, pictures, blue prints, and video information. 

•Potential targets are screened based on symbolic value and their potential to generate high 
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profile media attention. 
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Slide 25 
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(D 

Intelligence 

II and 

Surveillance 


• Targets showing potential receive a 
higher priority 

- Determine if target is beyond group’s 
capability. 

- Will it meet intended objectives? 

• Information gathered include 

- Praotioes/Procedures/Routines 

- Residence & Workplace 

- Transportation/Routes of Travel 

- Security Measures 

1-2-25 

Targets showing potential vulnerabilities are given a higher priority of effort. 

This priority establishes the requirement to gather additional information on the targets’ patterns 
over time. 
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Specific 
■ II Target 

Selection 


• Selection of a target for actual operational planning 
considers some of the following factors: 

- Does success affect a larger audience than the 
immediate victim(s)? 

- Will the target attract high profile media attention? 

- Does success make the desired statement to the 
correct target audience(s)? 

- Is the effect consistent with objectives of the group? 

- Does the target provide an advantage to the group 
by demonstrating its capabilities? 

- What are the costs versus benefits of conducting 
the operation? 

1-2-26 

No notes for this slide 
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Pre-attack 

IN Surveillance 

and Planning 


• Detailed and often long term surveillance is 
conducted 

- Conduct security studies. 

- Conduct detailed preparatory operations. 

- Recruit specialized operatives (if needed). 

- Procure a base of operations in the target 
area (safe houses, caches, etc.). 

- Design and test escape routes. 

- Decide on type of weapon or attack. 

1-2-27 

No notes for this slide 
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Attack plan is 
developed 

- Improve the 
odds of success 

- Confirm 
planning 
assumptions 

- Develop 
contingencies. 


1-2-28 


No notes for this slide 
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• Terrorists conducting pianned operations 
possess important tacticai advantages. 


Surprise 

Choice of time, place, and conditions of attack. 
Employment of diversions and secondary or 
follow-up attacks. 

Employment of security and support positions 
to neutralize target reaction forces and security 
measures. 


- It may be months before planned-for circumstances 
occur and the attack takes place 


1-2-29 


No notes for this slide 
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Escape plans are usually well rehearsed 
and executed. 

- Many terrorists want to survive the operation 
and escape. 

- Exception is suicide operation but support 
personnel may need to get-away 

Exploitation is the primary objective of the 
operation. 


1-2-30 

Media controi measures, prepared statements, and a host of other preparations are made to 
effectively exploit a successful operation. 

These will be timed to take advantage of media cycles for the selected target audiences. 
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Most Common Terrorist Threats 
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Most Common Terrorist Threats 

• Bombings (Suicide) • Ambushes 

• Arson • Armed Raid 

• Kidnapping/Extortion • Hostage Taking/ Barricade 

• Hijacking Situation 

• Skyjacking (Suicide) ’ Maimings 

• Assassination 

• Maritime operations 

• Standoff 

1-2-31 

1. First teaching point. Bombings are without a doubt the most prevaient way to accompiish 
a terrorist act and wiil remain so for the foreseeable future. 

2. Flostage taking incidents demand a higher degree of expertise and planning and are 
usually done only by groups that have evolved to a high degree of proficiency in other 
less sophisticated terrorist acts. We will discuss all these common terrorist acts in 
greater detail shortly. 

3. lED 

Slide32 

Choices 

Interaction type: Choices 

Passing score: 60 

Instructions: None 

Custom Message: None 

Learning Game Placeholder 

Learning Game: Choices 

Title: Choices 

V ) 

1. In this phase, the collection of information on a large number of potential targets, some of 
which may never be attacked, or seriously considered for attack. 

1) Intelligence and surveillance 

2) Target selection (Correct) 

3) Pre-attack 

4) Attack rehearsal 

2. During this phase security studies and design escape routes are conducted. 

1) Attack phase 

2) Pre-attack surveillance and planning (Correct) 

3) Target selection 

4) Delivery 

3. Most prevalent way to accomplish a terrorist act is: 

1) lED bomb (Correct) 

2) shoe bomb 

3) plane bomb 

4) kidnapping 
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Future Trends 


“Between now and 2015 terrorist tacties will become 
increasingly sophisticated and designed to achieve 
mass casualties.” 


National Intelligence Council 

"Global Trends 2015: A Dialogue About the Future With Nongovernmental 
Experts" Report (Dec 2000). 


Terrorism as a conflict method that has survived and evolved through several millennia to flourish 
in the modern information age, terrorism continues to adapt to meet the challenges of emerging 
forms of conflict, and exploit developments in technology and society. Terrorism has 
demonstrated increasing abilities to adapt to counter-terrorism measures and political failure. 
Terrorist are developing new capabilities of attack and improving the efficiency of existing 
methods. Additionally, terrorist groups have shown significant progress in escaping from a 
subordinate role in nation-state conflicts, and becoming prominent as international influences in 
their own right. They are becoming more integrated with other sub-state entities, such as criminal 
organizations and legitimately chartered corporations, and are gradually assuming a measure of 
control and identity with national governments. 
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No notes for this slide: 
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APPENDIX E. STORYBOARD EOR LESSON TWO. 
INSTRUCTIONAL MATERIALS 
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MOVES INSTITUTE 

Defense in Depth / Preplanned 

Responses 

Lesson Topic 

2.1 

No notes for this slide 
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Enabling Objectives 
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Enabling Objectives 

• STATE the purpose of Defense in Depth 

• DESCRIBE the principles of Defense in 

Depth 

• DESCRIBE the Assess, Warning and 

Threat Zones as used in the antiterrorism 
program concept of Defense in Depth 

• DESCRIBE a Naval Vessel Protection 

Zone (NVPZ) 

No notes for this slide 
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Enabling Objectives 

Duration: 00:00:13 
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Enabling Objectives 

'.1 

• DESCRIBE the purpose of pre¬ 
planned responses. 

• DESCRIBE the likely terrorist threats 
for developing pre-planned 
responses. 

No notes for this slide 
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Introduction 



“AT/FP training 
mnst be elevated to 
the same priority as 
primary mission 
training.” 


AT/FP!!!!!!” 


As a result of the commission on the investigation of the USS Cole attack it was recommended 
that “AT/FP training must be elevated to the same priority as primary mission training.” 

” We need to make AT/FP as important as damage control, Anti-Air warfare and other Navy 
primary missions. 
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Defense in Depth 


Purpose : To provide principles of 
Defense in Depth and associated 
concepts of the Navy Antiterrorism 
Program in order to link antiterrorism 
countermeasures with specified tasks. 


Defense in Depth 
a. Purpose 


(1) To provide principles of Defense in Depth and associated concepts of the Navy Anti- 
terrorism Program in order to link anti-terrorism countermeasures with specified tasks. 
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Defense in Depth 

Defense in Depth 



Duration: 00:00:24 


Advance mode: Auto 


Engaging the enemy at the earliest 
opportunity with security forces. 


• 

Employing weapons at maximum 
effective range. 


• 

Using blocking positions, obstacles, and 
supplementary positions throughout the 



area. 



Positioning and moving reserves and fire 
support units (Reaction Force). 


Conventional operations Defense in Depth is achieved by: 

(1) Engaging the enemy at the earliest opportunity with security forces. 

(2) Employing weapons at maximum effective range 

(3) Using blocking positions, obstacles, and supplementary positions throughout the area. 
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(4) Positioning and moving reserves and fire support units (Reaction Force). 
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Principles - Layered 
Defense 

Three Zones: 


- Assessment (Detect) 


- Warning (Deter) 


- Threat (Defend/Mitigate) 


The three principies of Layered defense 

(1) Assessment 

(2) Warning 

(3) Threat 
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Defense in Depth 

Assessment Zone - allows for initial ID of potential 
threat. 


‘We must separate the terrorists from the 
background noise. ’’ ADM Gehman 


Warning Zone - provides the area where the 
security force may respond with initial and 
escalating measures including warnings, 
challenges, use of non-lethal weapons if 
appropriate, and if time permits shouldering the 
threat. 

Threat Zone - engagement and destruction of the 
threat 


Assessment Zone 


(a) Ailows for initiai ID of potentiai threat. 

Warning Zone 

(a) Provides the area where the security force may respond with initiai and escaiating measures 
inciuding warnings, chaiienges, use of non-lethai weapons if appropriate, and if time permits 
shouidering the threat. 

Threat Zone 


(a) Engagement and destruction of the threat 
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Three Zones 

Duration: 00:00:12 
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ID, Track, Assess, and Neutralize 


“Set Conditions for the Use Of Force” 


The purposes of the three zones are to: 
“ID, Track, Assess, Neutralize” 

“Set Conditions for Use of Force” 
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NVPZ 

Duration: 00:01:16 
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The Naval Vessel Protection Zone uses the same principles of layered defense. 

The assessment zone, in the assessment zone contacts are identified by means of radar, radio 
communications or visual. 

The warning zone, in the warning zone contacts are given warnings and challenges. 

(a) Warnings are conducted in English and in the local language if in a foreign country. 

(b) Challenges are conducted in English and in the local language if in a foreign country. 

(c) Ensure that the vessel being challenge is aware that he is being challenged by Non-Lethal 
means. 

The threat zone, if the contact does not display compliance with the previous warnings the watch 
stander has to determine the threat of the contact. 

Determining the threat is done by determining the contacts capability, opportunity and intent. 
Capability refers to: Does the contact posses the capability to harm our forces? 

Opportunity refers to: Is there an opportunity for the contact to harm our forces? 

Intent refers to: Is the intent of the contact to harm our forces? 


109 

















Slidell « 
Inport Security 

Duration: 00:01:12 
Advance mode: Auto 


The Naval Vessel Protection Zone and the Defense in depth concept also applies to ships in-port 
The Extended security zone - delineated by the Dashed Blue line corresponds to the assessment 
zone. 

Assets used in this area are: 

(a) Shore side: 

HMMVs or patrol cars 

Armed Personnel with Id checks at the gates. 

(b) Sea side 
Communications 
Radar equipment 
Small Boats 

The Perimeter security zone - delineated by the Solid green corresponds to the warning zone. 
Assets used in this area are: 

(a) Shore side 

Armed Pier sentries at the ECPs 

(b) Sea side 
Small Boats 
Sea Barriers 

The External security zone - delineated by the solid yellow line corresponds to the threat zone. 
Assets used in this area are: 

(a) Shore side 
Armed Pier rovers 
Armed Topside rovers 

100% illumination of pier areas 

(b) Sea side 
Small Boats 
Sea Barriers 

Armed Topside rovers 

100% illumination of ship boundaries with the water _ 



Inport Security 


Extended Security 

- Random Patrols 

- Personal Arms 

- Communicatons 

- Small Boats 

• Patrol cars 

• Radars 

Observation Post 

- Personal Arms 

- Communications 

- Mobile 

- Night Vision 

Perirrgter Security 

Zone 

- Day / Nght 
Observation 

- Small Boats 

- Sea Barriers 

- Personal Arms 



External Security 

- Armed Sentries 
•SMGs 

• Riot Control 
Agents 

- Hardened 
Barricades 

- Armed Sentries 
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High Vaiue Assets 

High vaiue assets are those assets that the enemy wants to target because: 

The target’s symboiic vaiue 

The target’s potentiai to generate high profile media attention 
The target is the objective of the terrorist group 

This picture was taken prior to the USS Cole attack, security was not our number one priority. 
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ElfTERI NG TERRI TORI AL WATERS 



The Naval Vessel Protection Zone and the Defense in depth concept also applies to ships 
transiting in or out of port and in narrow channe s. 
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Pre-Planned Reponses 
(PPR) 

PPRs are developed, exercised actions 
and measures that are implemented to: 

- Track 

- Assess 

- Neutralize terrorist threats 

PPRs are the tactics, techniques used 
to carry out the AT program concept of: 

- Detect 

- Deter 

- Defend 

- Mitigate 


Pre-Planned Reponses (PPR’s) are developed, exercised actions and measures that are 
implemented to: 

a. Track 

b. Assess 
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c. Neutralize terrorist threats 

PPR’s are the tactics, techniques used to carry out the AT program concept of: 

(1) Detect 

(2) Deter 

(3) Defend 

(4) Mitigate 
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Pre-Planned Reponses 


Pre-Planned Reponses (PPR) 

(PPR) 

Cil ” 

Duration: 00:00:31 

Advance mode: Auto 

• 

Likely threats: 



- Small boats/Deep draft vessels 

- Swimmers/Mines 

- Personnel-borne lEDs 

- Vehicle-born lEDs 

- Rail-born lEDs 

- Aircraft 

- Standoff attacks 



- Man-portable defense systems 
(MANPADS) 



- CBNE attacks (covert-overt) 


Likely threats to U.S. Assets 

(1) Small boats/Deep draft vessels 

(2) Swimmers/Mines 

(3) Personnel borne lEDs 

(4) Vehicle-born lEDs 

(5) Rail-born lEDs 

(6) Aircraft 

(7) Standoff attacks 

(8) Man-portable defense systems (MANPADS) 

(9) CBRNE attacks (covert-overt) 
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Pre-Planned Reponses (PPR) 

Duration: 00:00:23 
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• PPRs are inherent in your COA 
development 




Pre-Planned Reponses 
(PPR) 


- During war gaming each friendiy COA is 
examined against seiected threat COAs 

- If a certain threat attacks or endangers the 
command, it is essential to know how the 
command will react with PPRs. 


• Sufficient 


• Timely 


PPR’s are inherent in your course of action (COA) Development 

(1) During war gaming each friendly COA is examined against selected threat COAs 

(2) If a certain threat attacks or endangers the command, it is essential to know how the 
command will react with PPR’s 

(a) Sufficient 

(b) Timely 


112 












Pre-Planned Reponses 
(PPR) 

Pre-Planned Responses 

- NTTP 3-07.2.1, Chapter 6 

- Good check list of PPRs 

- Must look beyond check lists 

Must think critically and dynamically 
about potential threats for each 
mission and for each vulnerability. 


Pre-Planned Responses 

(1) NTTP 3-07.2.1, Rev A, Chapter 6 

(2) Good check list of PPR’s 

(3) Must look beyond check lists 

(4) Must think critically and dynamically about potential threats for each mission and for each 


vulnerability 
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Set up defense in depth for the 


NAVSTA scenario and iist and 


observe the pre-pianned responses 

Set up defense in depth 

for boat-born lED. 

for the NAVSTA scenario 

Duration: 00:00:13 

and iist and observe the 

Advance mode: Auto 

pre-pianned responses 
for boat-born lED. 
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APPENDIX F. STUDY I PRE-TEST WITH ANSWERS 


NAME 


Pre-Test: 


True/False, 5 points, 1 attempt(s) permitted. 

1. In accordance to the Department of Defense terrorism is defined as: 

The calculated use of violence or threat of violence to inculcate fear; intended to 
coerce or to intimidate governments or societies in the pursuit of goals that are 
generally political, religious or ideological”. 

(In accordance to the Department of Defense terrorism is defined as: The 
calculated use of violence or threat of violence to inculcate fear; intended to coerce 
or to intimidate governments or societies in the pursuit of goals that are generally 
political, religious or ideological”. ) 


Correct 

Choice 

X 

True 

False 


Tme/False, 5 points, 1 attempt(s) permitted. 

2. The categories of terrorist organizations are: 

government affiliated, motivation, ideological (The categories of terrorist 
organizations are: government affiliated, motivation, ideological) 


Correct 

Choice 

X 

True 

False 


Multiple Response, 5 points, 1 attempt(s) permitted. 

3. Multiple response question: 

Government affiliated terrorist organization are comprised of the following: 
(Multiple response question: Government affiliated terrorist organization are 
comprised of the following:) 


Correct 

Choice 

X 

non-state supported 

1 Religious | 

X 

state-supported 

1 Government | 

X 

state-directed 


Multiple Choice, 5 points, 1 attempt(s) permitted. 
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4. Motivation affiliated terrorist organization are comprised of the following except: 


Correct Choice _ 

_ Separatist _ 

_ ethnocentric groups 

X Ideological _ 

_ Nationalistic _ 

_Revolutionary 


Multiple Response, 5 points, 1 attempt(s) permitted. 

5. Multiple response question: 

Ideological affiliated terrorist organization are comprised of the following except: 
(Multiple response question: Ideological affiliated terrorist organization are 
comprised of the following except:) 


Correct Choice 

Political 

X Nationalist 

Religious 

social orientation 

X Terrorism 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

6. Political affiliated terrorist organization are comprised of the following: 

Correct Choice 

X Right wing, Left wing, Anarchist 

Right wing. Left wing. Center 

Center, Left wing. Anarchist 

Left wing. Anarchist, Center 

Right wing. Center, Anarchist 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

7. Terrorist hope to accomplish the following goals except: 

Correct Choice 

Recognition 

Vengeance 

Weaken government 

X create their own government 

discourage foreign investment 
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True/False, 5 points, 1 attempt(s) permitted. 

8. The definition of internal terrorist training is: conducted at the cell level hy 
members of the group within the target country, usually in a clandestine fashion. 
(The definition of internal terrorist training is: conducted at the cell level by 
members of the group within the target country, usually in a clandestine fashion.) 


Correct 

Choice 

X 

True 

False 


Multiple Choice, 5 points, 1 attempt(s) permitted. 

9. These training programs are included in a typical terrorist training program 
except: 

(These training programs are included in a typical terrorist training program 
except: ) 


Correct Choice 

Physical condition 

Weapons training 

Political indoctrination 

Combat tactics 

X Capital investment 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

10. Which one of these is a characteristic of a terrorist group: 

Correct Choice 

Overt 

Military strong 

Based outside urban areas 

Poor training 

X Highly mobile 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

11. The first stage of a terrorist attack is: 

Correct Choice 

X Target selection 

Intelligence and surveillance 

Pre-attack 

Specific target selection 

Attack rehearsal 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

12. In this phase, Targets showing potential vulnerabilities are given a higher 
priority of effort. This priority establishes the requirement to gather additional 
information on the targets’ patterns over time. 


Correct Choice 

Target selection 

X Intelligence and surveillance 

Pre-attack 

Specific target selection 

Attack rehearsal 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

13. Once a target is selected the following surveillance is conducted except: 

Correct Choice 

Conduct security studies 

Recruit specialized operatives 

Design and test escape routes 

Decide on attack weapon 

X Determine the objective of the group 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

14. The last step of a terrorist attack is: 

Correct Choice 

X Escape and exploitation 

Attack 

Actions on the objective 

Attack rehearsal 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

15. All of these are common terrorist threats except: 

Correct Choice 

Bombing 

Arson 

Ambushes 

Maiming 

X War 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

16. Based on what we learned on this course a future terrorist tactic may include: 


Correct Choice 

Computer attack 

Bombs delivered by animals 

Shoe bombs 

X CBRNE 

Eiquid bombs on airplanes 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

17. Terrorist will be looking to achieve this type of casualty by the year 2015: 

Correct Choice 

Network casualties 

Economy casualties 

X Mass casualties 

Political casualties 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

18. These are tactical advantages of a terrorist conducting an attack except: 

Correct Choice 

Choice of time 

Choice of place 

X Numbers of security forces 

Surprise 

Diversions for follow up attacks 


Multiple Choice, 5 points, 1 attempt(s) permitted. 

19. These type of groups are dedicated to the overthrow of an established order and 
replacing it with a new political or social structure. (These type of groups are 
dedicated to the overthrow of an established order and replacing it with a new 
political or social structure.) 


Correct Choice _ 

_ Separatist group 

_ Ethnocentric group 

X Revolutionary group 

_Political group_ 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

20. These type of groups operate as an agent of a government and receive 
substantial 

intelligence, logistic, and operational support from the sponsoring government. 
(These type of groups operate as an agent of a government and receive substantial 
intelligence, logistic, and operational support from the sponsoring government.) 
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APPENDIX G. STUDY I POST-TEST WITH ANSWERS 


NAME_ Pre-Test: 

Trae/False, 5 points, 1 attempt(s) permitted. 

1. The following elements are essential in the definition of terrorism in accordance 
to the Department of Defense: 

violence or threat of violence, fear, coercion, pursuit of goals (The following 
elements are essential in the definition of terrorism in accordance to the Department 
of Defense: violence or threat of violence, fear, coercion, pursuit of goals) 


Correct 

Choice 

X 

True 

False 


Multiple Response, 5 points, 1 attempt(s) permitted. 

2. Multiple response question: 

Which is/are categories of terrorist organization? (Multiple response question: 
Which is/are categories of terrorist organization?) 


Correct 

Choice 

X 

government affiliated 

state governed 

X 

Motivation 

1 Separatist | 

X 

Ideological 


Multiple Response, 5 points, 1 attempt(s) permitted. 

3. Which is an indication of a government affiliated terrorist organization? 


Correct Choice 


X state support 


Violence 


Threat 


Fear 


Coercion 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

4. These are included in a government affiliated terrorist organization except: 


Correct Choice 

Separatist 

ethnocentric groups 

Nationalistic 

Revolutionary 

X Political 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

5. Which is included in a ideological affiliated terrorist organization? 

Correct Choice 

X Religious 

Rationalist 

Nationalist 

Anarchist 


Multiple Response, 5 points, 1 attempt(s) permitted. 

6. Multiple response question: 

Which is/are included in a political affiliated terrorist organization? (Multiple 
response question: Which is/are included in a political affiliated terrorist 
organization?) 


Correct 

Choice 

X 

right wing 

X 

left wing 

X 

Anarchist 

Religious 

Radicalism 


Multiple Choice, 5 points, 1 attempt(s) permitted. 

7. Which one of these is a goal terrorists do not want to achieve? 


Correct Choice _ 

Eliminate western influence, recognition 

_ Free prisoner, satisfy vengeance _ 

_ Influence government decisions, disrupt government 

X Generate money, create their own government_ 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

8. Which 2 training programs are not in a typical terrorist training program. 
(Which 2 training programs are not in a typical terrorist training program. ) 


Correct Choice _ 

_ Physical condition, survival skills _ 

_ Communications, political indoctrination 

_ Weapons, explosive training _ 

X Electronic, network warfare_ 


True/False, 5 points, 1 attempt(s) permitted. 

9. In accordance with the DoD the definition of external terrorist is: 

Non-group members outside the operational area conduct external training. It can 
take two different forms. First, the operatives may travel abroad, individually or as 
a group, to training camps or to other permanent institutions for their training. 

(In accordance with the DoD the definition of external terrorist is: Non-group 
members outside the operational area conduct external training. It can take two 
different forms. First, the operatives may travel abroad, individually or as a group, 
to training camps or to other permanent institutions for their training. ) 


Correct 

Choice 

X 

True 

False 


Multiple Choice, 5 points, 1 attempt(s) permitted. 

10. Which one of these is a characteristic of a terrorist group: 


Correct Choice 


X 


Covert 


Urban based 


Well trained 


Overt 


Military weaker 


Multiple Choice, 5 points, 1 attempt(s) permitted. 

II. In this phase, the collection of information on a large number of potential 
targets, some of which may never be attacked, or seriously considered for attack. 


Correct Choice _ 

X Target selection _ 

_ Intelligence and surveillance 

Pre-attack 

_ Specific target selection 

Attack rehearsal 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

12. Selection of a target in the target selection stage, for actual operational planning 
considers some of the following factors except: 


Correct Choice 

Will the target attract high profile media attention? 

Does success make the desired statement to the correct target audience(s)? 

Is the effect consistent with objectives of the group? 

the target provide an advantage to the group by demonstrating its capabilities? 

X Are escape routes available? 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

13. During this phase security studies and design escape routes are conducted. 

Correct Choice 

Attack phase 

X Pre-attack surveillance and planning 

Target selection 

Attack 

Attack rehearsal 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

14. Which one of these is not a stage of the terrorist planning cycle. 

Correct Choice 

Target selection 

Intelligence and surveillance 

Attack 

Actions on the objective 

X Communication with media 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

15. Most prevalent way to accomplish a terrorist act is: 

Correct Choice 

Kidnapping 

X lED bombings 

Carjacking 

Plane hijacking 

Arson 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

16. The following are future trends of terrorist tactics except: 


Correct Choice 

CBRNE 

lED bombs 

X UAV delivered bombs 

High yield explosives 

Nuclear dirty bomb 

Multiple Choice, 5 points, 1 attempt(s) permitted. 

17. The primary objective of a terrorist plan is: 

Correct Choice 

X Exploitation 

Economic success 

People anger 

Suicide for afterlife rewards 


Multiple Choice, 5 points, 1 attempt(s) permitted. 

18. These group objective is the separation from existing entities through 
independence, political autonomy, or religious freedom or domination. Their 
ideologies suhscrihe to include social justice or equity, anti-imperialism, as 
well as the resistance to conquest or occupation hy a foreign power. 

(These group objective is the separation from existing entities through 
independence, political autonomy, or religious freedom or domination. Their 
ideologies subscribe to include social justice or equity, anti-imperialism, as well as 
the resistance to conquest or occupation by a foreign power.) 


Correct Choice _ 

X Separatist group 

_ Ethnocentric group 

_ Revolutionary group 

_Political group_ 
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Multiple Choice, 5 points, 1 attempt(s) permitted. 

19. These are terrorist groups that operate autonomously, receiving no 
significant support from any government. 

(These are terrorist groups that operate autonomously, receiving no significant 
support from any government.) 


Correct Choice 


State supported 


X Non-state supported 


State directed 


Proxy army 


Multiple Choice, 5 points, 1 attempt(s) permitted. 

20. This is the definition of a deliberate terrorist element. 


Correct Choice _ 

X Terrorism is an activity planned and intended to achieve particular goals. 

_ Terrorist groups demand change, revolution or political movement. _ 

_ The intended result is to create a psychological terror _ 

Violence, coercion, and destruction are used in the commission of the act. 

Violence, coercion, and destruction are used in the commission of the act. 
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APPENDIX H. STUDY II PRE-TEST WITH ANSWERS 


NAME_ Pre-Test: 

Pre-Test objectives: 

1. Measure the student’s knowledge about the following subjects: 

• Defense in Depth 

• Assessment, Warning and Threat Zones as used in the anti-terrorism program 

• Naval Vessel Protection Zone (NVPZ) 

• Pre-planned responses. 


1 ■ What is the purpose of defense in depth? 



a. To provide the ATO with pre-planned responses in case of terrorist attacks. 

X 

b. To provide principles of Defense in Depth and associated concepts of the Navy 
Anti-terrorism Program in order to link anti-terrorism countermeasures with 
specified tasks. 


c. To engage the enemy at the earliest opportunity. 


d. Employ weapons at the maximum effective range. 


2. Which answer contains 2 principles of defense in depth? 



a. The use of small boats and shipboard security forces. 


b. Engage enemy forces at the earliest opportunity to prevent further damage. 

X 

c. Engaging the enemy at the earliest opportunity, employ weapons at maximum 
effective range. 


d. using obstacles throughout the area and small boats. 


3. What are the principles of layered defense as used in 
Anti Terrorism Force Protection (AT/FP)?_ 


X 

a. Assessment, warning and threat. 


b. Assessment, warning and capability opportunity intent (COI). 


c. Assessment, threat and COI. 


d. none of the above. 


4. What is the proper definition for assessment zone; 


X 

a. Allows for initial ID of potential threat. 


b. Provides the area where the security force may respond with initial and 
escalating measures including warnings, challenges, use of non-lethal weapons if 
appropriate, and if time permits shouldering the threat. 
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c. Engagement and destruction of the threat. 


d. Area where capability, opportunity and intent are determined. 


5. What is the proper definition for warning zone; 



a. Allows for initial ID of potential threat. 

X 

b. Provides the area where the security force may respond with initial and 
escalating measures including warnings, challenges, use of non-lethal weapons if 
appropriate, and if time permits shouldering the threat. 


c. Engagement and destruction of the threat. 


d. Area where capability, opportunity and intent are determined. 


6. What is the proper definition for threat zone; 



a. Allows for initial ID of potential threat. 


b. Provides the area where the security force may respond with initial and 
escalating measures including warnings, challenges, use of non-lethal weapons if 
appropriate, and if time permits shouldering the threat. 

X 

c. Engagement and destruction of the threat. 


d. Area where capability, opportunity and intent are determined. 


7. What is the purpose of the Assessment, warning and threat zones? 



a. Neutralize terrorist contacts. 


b. Warn unidentified vessels of the Naval Vessel Protection Zone. 

X 

c. ID, Track, Assess, Neutralize and Set Conditions for Use of Eorce. 


d. Allows for initial ID of potential threat. 


8. How is the Naval Vessel Protection Zone organized? 



a. ID 


b. Warn 


c. Assessment 

X 

d. threat 


9. What asset is used in the Assessment zone? 


X 

a. Patrol craft. 


b. Air search radars. 


c. Intelligence briefs. 


d. NCIS 
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10. Which actions the watch-stander must take when an unidentified vessel enters 
the warning zone? _ 



a. Ensure that the vessel being challenge is aware that he is being challenged 


b. Small arm fire across their bow. 


c. Use the patrol boat to shoulder the contact. 

X 

d. Verbal Warnings in English and in the local language if in a foreign country. 


11. Which criteria the watch standee uses to determine threat? 



a. Opportunity 


b. Capability 


c. Intent 

X 

d. threat 


12. What is the purpose of Pre-planned responses? 



a. Rehearse terrorist attacks. 

X 

b. Track, assess and neutralize terrorist threats. 


c. Plan for terrorist possible actions. 


d. Neutralize terrorist threats. 


13. Can Pre-Planned Responses be modified to tailor specific situations? 


X 

a. True 


b. Ealse 


14. Multiple answer question 

Select (3) likely threats that any command has to be prepared for? 


X 

a. Small boats/Deep draft vessels 

X 

b. Swimmers/Mines 


c. submarine attacks. 


d. WMD. 

X 

e. CBRNE attacks (covert-overt) 


15. How are Pre-Planned Responses used in AT/FP program? 



a. Used to prevent terrorist attacks. 


b. Used in after action reports to evaluate incidents. 

X 

c. During war gaming each friendly COA is examined against selected threat 
COAs. 


d. check list. 
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APPENDIX I. STUDY II POST-TEST WITH ANSWERS 


NAME_ Post-Test: 

Post-Test objectives: 

1. Measure the student’s knowledge about the following subjects: 

• Defense in Depth 

• Assessment, Warning and Threat Zones as used in the anti-terrorism program 

• Naval Vessel Protection Zone (NVPZ) 

• Pre-planned responses. 


1 ■ What is the purpose of defense in depth? 



a. To provide the ATO with pre-planned responses in case of terrorist attacks. 

X 

b. To provide principles of Defense in Depth and associated concepts of the Navy 
Anti-terrorism Program in order to link anti-terrorism countermeasures with 
specified tasks. 


c. To engage the enemy at the earliest opportunity. 


d. Employ weapons at the maximum effective range. 


2. Are Pre-Planned Responses used in Course of Action development? 


X 

a. True 


b. Ealse 


3. What does a good AT/FP Officer must do to ensure that he assesses all the 
possible terrorist threats? _ 



a. Ask a lot of questions. 


b. Use check lists only. 

X 

c. Must think critically and dynamically about potential threats for each mission 
and for each vulnerability. 


d. Practice, practice, practice. 


4. Which are the 3 principles of layered defense as used in 
Anti Terrorism Force Protection (AT/FP)?_ 


X 

a. Assessment, warning and threat. 


b. Assessment, warning and capability opportunity intent (COI). 


c. Assessment, threat and COI. 


d. none of the above. 
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5. What is the proper definition for assessment zone; 


X 

a. Allows for initial ID of potential threat. 


b. Provides the area where the security force may respond with initial and 
escalating measures including warnings, challenges, use of non-lethal weapons if 
appropriate, and if time permits shouldering the threat. 


c. Engagement and destruction of the threat. 


d. Area where capability, opportunity and intent are determined. 


6. What is the proper definition for warning zone; 



a. Allows for initial ID of potential threat. 

X 

b. Provides the area where the security force may respond with initial and 
escalating measures including warnings, challenges, use of non-lethal weapons if 
appropriate, and if time permits shouldering the threat. 


c. Engagement and destruction of the threat. 


d. Area where capability, opportunity and intent are determined. 


7. What is the proper definition for threat zone; 



a. Allows for initial ID of potential threat. 


b. Provides the area where the security force may respond with initial and 
escalating measures including warnings, challenges, use of non-lethal weapons if 
appropriate, and if time permits shouldering the threat. 

X 

c. Engagement and destruction of the threat. 


d. Area where capability, opportunity and intent are determined. 


8. What is the purpose of the assessment, warning and threat zones? 



a. Neutralize terrorist contacts. 


b. Warn unidentified vessels of the Naval Vessel Protection Zone. 

X 

c. ID, Track, Assess, Neutralize and Set Conditions for Use of Eorce. 


d. Allows for initial ID of potential threat. 


9. How is the Naval Vessel Protection Zone organized? 



a. ID 


b. Warn 


c. Assessment 

X 

d. Assessment 
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10. What asset is used in the Assessment zone? 


X 

a. Patrol craft. 


b. Air search radars. 


c. Intelligence briefs. 


d. NCIS 


11. Which criteria does the watch-stander uses to determine threat? 



a. Opportunity 


b. Capability 


c. Intent 

X 

d. HVU 


12. What is the purpose of Pre-planned responses? 



a. Rehearse terrorist attacks. 

X 

b. Track, assess and neutralize terrorist threats. 


c. Plan for terrorist possible actions. 


d. Neutralize terrorist threats. 


13. Can Pre-Planned Responses he modified to tailor specific situations? 


X 

a. True 


b. False 


14. Multiple answer question 

Select (3) likely threats that any command has to be prepared for? 


X 

a. Small boats/Deep draft vessels 

X 

b. Swimmers/Mines 


c. submarine attacks. 


d. WMD. 

X 

e. CBRNE attacks (covert-overt) 


15. How are Pre-Planned Responses used in AT/FP program? 



a. Used to prevent terrorist attacks. 


b. Used in after action reports to evaluate incidents. 

X 

c. During war gaming each friendly COA is examined against selected threat 
COAs. 


d. check list. 
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APPENDIX J. SAMPLE PORT BRIEEING INEORMATION 

SHEET 


INFORMATION SHEET 1 
TASKS 


10 May 2007. 


From: Commanding Officer 


To: Designated Anti-terrorism Officer 


Subj: IN PORT SECURITY PLAN ICO ANY PORT FOR A PORT VISIT 


1. Two days from now, at approximately 2000 Local, we will be arriving in any port for a (4) day port visit. We will depart at 
0600 Local on the fourth day. 


2. The Freedom Fighters United (FFU) is an Islamic fundamentalist based group consisting of approximately 10 members with 
ties to Hizballah and HAMAS terrorist organizations operating in the Middle East. FFU conducts their operations covertly 
primarily in and around the city of nay city, any state. Operatives have been observed in the past around the area of NAVSTA 
nay NAVSTA to what appears conducting surveillance. With the exception of protests at the base gates in April 2007 by 
members of the FFU, there are no indications of preparations for an attack by any organization to military assets on the base. FBI 
reports that the FFU have recently purchased 2 small boats. Last week an Army explosives DEPOT was broken into and large 
quantities of explosives were stolen. FBI is investigating but preliminary reports point to the FFU. 


3. Any NAVSTA is currently in FPCON Bravo. 


4. TASKING: 


a. You area directed to review directives and instructions regarding AT/FP planning and In Port Security Plans (ISP). 


b. Given 2 RHIBS, pier radar and barriers use the AFTP SavageStudio to develop an asset deployment and use given pre 
planned responses against explosive laden vessels. 


135 




THIS PAGE INTENTIONALLY LEET BLANK 


136 



APPENDIX K. INSTITUIONAL REVIEW BOARD (IRB) EORMS 


Naval Postgraduate School 

Informed Consent Form 

Introduction. You are invited to participate in a study entitled Training effectiveness of 
Force Protection Training using Computer-based Instruction with embedded Simulation 
being conducted by the Naval Postgraduate School MOVES Institute. 

Procedures. You will be asked to complete a questionnaire with questions about your 
rank, rate and assignments previous and current. You will conduct a pre-test about Force 
protection knowledge, that way we can measure your base knowledge. Second you will 
complete a CBT lesson on Force Protection, and then you will complete a Force 
Protection scenario. The conclusion of the research will be after completion of the post¬ 
test, to measure if any knowledge has been acquired. Total time required 45 minutes. 

Risks and Benefits. I understand that this project does not involve greater than minimal 
risk and involves no known reasonably foreseeable risks or hazards greater than those 
encountered in everyday life. I have also been informed of any benefits to myself or to 
others that may reasonably be expected as a result of this research. 

Compensation. I understand that no tangible compensation will be given. I understand 
that a copy of the research results will be available at the conclusion of the experiment via 
e-mail and thesis results will be available at the library. 

Confidentiality & Privacy Act. I understand that all records of this study will be kept 
confidential and that my privacy will be safeguarded. No information will be publicly 
accessible which could identify me as a participant. I will be identified only as a code 
number on all research forms/data bases. My name on any signed document will not be 
paired with my code number in order to protect my identity. I understand that records of 
my participation will be maintained by NPS for three years, after which they will be 
destroyed. 

Voluntary Nature of the Study. I understand that my participation is strictly voluntary, 
and if I agree to participate, I am free to withdraw at any time without prejudice. 

Points of Contact. I understand that if I have any questions or comments regarding this 
project upon the completion of my participation, I should contact the Principal Investigator, 
, Dr. Don Brutzman, 656-2149, brutzman@nps.edu. Any medical questions should be 
addressed to FTC Eric Morgan, MC, USA, (CO, POM Medical Clinic), (831) 242-7550, 
eric.morgan@nw.amedd.army.mil. Any other questions or concerns may be addressed to 
the IRB Chair, FT Brent Olde, 656-3807, baolde@nps.edu . 
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Statement of Consent. I have been provided with a full explanation of the purpose, 
procedures, and duration of my participation in this research project. I understand how 
my identification will be safeguarded and have had all my questions answered. I have 
been provided a copy of this form for my records and I agree to participate in this study. I 
understand that by agreeing to participate in this research and signing this form, I do not 
waive any of my legal rights. 

Participant’s Signature Date 

Researcher’s Signature Date 
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APPENDIX L. STUDENT DEMOGRAPHICS QUESTIONNAIRE 


NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 

Transfer of Training Study between Classroom-Based and Computer-Based 

Training 

Questionnaire: 

1. Student demographics: 

a. What is your rank? 

b. What is your rate if applicabie? 

c. gender 

d. age 

e. History of assignments: 

f. What is your total time in the Navy? 

g. Do you have any previous knowledge of ATFP or attended any other ATFP courses? 

h. In a scale of 1-5. 1. no knowledge at all 2. little knowledge 3. basic knowledge 4. reasonable 
knowledge 5.expert knowledge. How do you rate your ATFP knowledge? 

I. How often does your command conduct ATFP training? 
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APPENDIX M. SAVAGESTUDIO EXERCISE STUDENT TASK 

CHECKLIST 


SAVAGESTUDIO EXERCISE CHECKLIST 

Date:_ Student #:_ 

Training Objectives: 

1. Develop and implement Pre-Planned Responses that will be effective mechanisms to 

detect, deter, defend and mitigate likely terrorist threats against U.S. Navy assets and 

installations. 

Enabling Objectives: 

2.3b Use the SavageStudio visual mode to demonstrate the assessment, warning and 
threat zones as used in the anti-terrorism program concept of defense in depth. 
(Knowledge) Performance data for this objective will be collected using the 
SavageStudio, student will explain or address the 3 zones as they apply to the 
scenario. 

2.3c Use the SavageStudio in visual mode to demonstrate the application of the 
Assessment, Warning and Threat Zones as used in the anti-terrorism program. 
(Skill) Performance data for this objective will be collected using the 
SavageStudio, student will demonstrate or identify the 3 zones as they apply to 
the scenario. 

2.5a Use the SavageStudio in visual mode to identify critical assets, and vulnerabilities 
for the NAVSTA scenario. (Knowledge) Performance data for this objective will 
be collected using the SavageStudio, describe and explain the assessment of the 
environment and situation, the student will explain potential threats and 
vulnerabilities as they apply to the scenario. 

2.5b, Use the SavageStudio in visual mode to explain the anti-terrorism environment 
and situation; using the X3D visual scene of the generated scenario explain the 
force protection employment of assets to protect given critical assets while 
minimizing vulnerabilities. (Skill) Performance data for this objective will be 
collected using the SavageStudio, the student will identify potential threats and 
vulnerabilities as they apply to the scenario. 
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Task 2 SAT UNSAT 


Set up Defense in Depth for the NAVSTA Scenario 

N/A 

N/A 

Assessment zone established or addressed? 



Warning zone established or addressed? 



Threat zone established or addressed? 



List the pre-planned responses for boat-born lED. 



Asses the environment and situation 

N/A 

N/A 

Identify critical assets 



Identify vulnerabilities 



Exercise scenario to describe the principles of defense in depth 
as they occur in the scenario. 



Observe the pre-planned responses for boat-born lED. 
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APPENDIX N. ANALYST REPORT 


A. ANALYST REPORT INTERFACE 

The following figures illustrate the Viskit interface used for creating the analyst report. A 
brief description is provided for each figure. 

1. The Heading Panel is used to annotate the report information. Information 
required is title of the report, author’s name, date, time and classification of the report. 



Figure 38. Viskit Analyst Report Editor displaying the Heading Panel window. 
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2. The Executive Summary allows the analyst to provide a synopsis of the 
simulation. It provides the context for the remainder of the report and ensures that 
someone reading the report has an overview of the entire simulation problem 



Figure 39. SavageStudio scenario generation display window depicting scenario loading 

from the scenario run library. 

3. Simulation Location Panel is used to provide information about the simulation 
environment. The comments in this section allow before and after simulation 
comparisons from the analyst point of view. 





Figure 40. 


SavageStudio scenario generation display window depicting scenario loading 
from the scenario run library. 
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4. The Simulation Configuration panel is used to enumerate all of the entities and 
objects employed in the simulation. A table of SimEntities is auto-generated from the 
assembly output file which lists the description of the entity as well as the name of the 



Figure 41. SavageStudio scenario generation display window depicting scenario loading 

from the scenario run library. 


5. The Entity Parameters panel allows the analyst to make annotations about the 
parameters used in the simulation. The panel provides the ability to preview the 
parameters that will be incorporated into the simulation. Analysts are required to point 
out parameters that were used to influence the outcome of the simulation. 



Figure 42. SavageStudio scenario generation display window depicting scenario loading 

from the scenario run library. 
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6. The Behavior Definitions panel allows before and after annotations regarding the level 
of detail about behavior definition descriptions. The Analysts are required to provide a 
description about the behaviors used in the simulation. 



Figure 43. SavageStudio scenario generation display window depicting scenario loading 

from the scenario run library. 

7. The Statistical Results panel allows a user to provide before and after 
comments about the replication and summary statistics for a specific simulation. The 
analyst has the option whether or not he will like to include histogram charts of the 
replication statistics as part of their output. 



Figure 44. SavageStudio scenario generation display window depicting scenario loading 

from the scenario run library. 
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8. The Conclusions and recommendations panel allows the analysts to use the 
interface to provide comments regarding the results of the simulation and 
recommendations for future work. 



Figure 45. SavageStudio scenario generation display window depicting scenario loading 

from the scenario run library. 
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B. ANALYST REPORT EXAMPLE 


THIS REPORT IS: UNCLASSIFIED 

AT/FP training analysis for the Bremerton 

scenario 

Analyst: Participant number 20 
Analysis date: 7/22/07 1:21 PM 3/19/07 2:17 PM 


Executive Summary 

The purpose of this report is to analyze the participants force protection implementation 
and security assets employment given a specific threat as identified in the task sheet. The 
participant reviews the generated report and analyze the results and deter mi nes mission 
success. If the mission is not successful the participant returns to the SavageStudio 
application to modify asset employment. 


Simulation Location 

Description of Location Features. The scenario generated by this simulation will be of 
Bremerton Harbor by participant number 20. 


Post-Experiment Analysis of Significant Location Features. Participant number 20 
completed 83% of training objectives. He did not properly assess the port vulnerabilities. 
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Assembly Configuration for Viskit Simulation 

Assembly Design Considerations. Assets used by the participant included: 1 Nimitz 
Carrier, 2 explosive-laden vessels, 2 Rhib patrol boats and Barriers 

Post-Experiment Analysis of Simulation Assembly Design. Barrier employment and patrol 
areas assignment affect the explosive-laden vessels attack profile. 


. .‘Hi*. > 

> .* w 
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Simulation Entities 


Entity Name 

Behavior Definition 

PierBravo 

Obstacles.Pier 

PierCharlie 

Obstacles.Pier 

PierDelta 

Obstacles.Pier 

MooringE 

Obstacles.Pier 

MooringE 

Obstacles.Pier 

MooringG 

Obstacles.Pier 

MooringBuoyA 13 

Obstacles.Buoy 

MooringBuoyA 12 

Obstacles.Buoy 

MorringBuoyAl 1 

Obstacles.Buoy 

MooringBuoyLl 

Obstacles.Buoy 

MooringB uoyL2 

Obstacles.Buoy 

MooringB uoyL3 

Obstacles.Buoy 

MooringB uoyL4 

Obstacles.Buoy 
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T erroristCellPlanner 

Hostile.TerroristCellPlanner 

Terrorist 1 

Hostile.ExplosiveEadenVessel 

Terrorist2 

Hostile.ExplosiveEadenVessel 

CVN70 

Eriendly. Military Ship 

PierBravo 

Obstacles.Pier 

PierCharlie 

Obstacles.Pier 

PierDelta 

Obstacles.Pier 

MooringE 

Obstacles.Pier 

MooringE 

Obstacles.Pier 

MooringG 

Obstacles.Pier 

RHIB 

Eriendly .MilitaryPatrolCraft 

NauticalChart 

Utilities.NauticalChart 

MooringBuoyA 13 

Obstacles.Buoy 

MooringBuoyA 12 

Obstacles.Buoy 

MorringBuoyAl 1 

Obstacles.Buoy 

MooringBuoyEl 

Obstacles.Buoy 

MooringB uoyE2 

Obstacles.Buoy 

MooringB uoyL3 

Obstacles.Buoy 

MooringB uoyE4 

Obstacles.Buoy 


Entity Parameters 

Entity Parameters Overview. Entity parameters are initialization values used to define 
new event graphs. These values are pulled directly from the assembly. 


Behavior Definitions 

Description of Behavior Design. The following table specifies the specific parameters for 
entities and objects in this simulation. 

Post-Experiment Analysis of Entity Behaviors. The following table specifies the specific 
parameters for entities and objects in this simulation. 


Statistical Results 

Description of Expected Results. 2 replications were evaluated for statistical purposes. 

Analysis of Experimental Results. Comments Participants are allowed to review their 
statistical results and encouraged to alter the scenario with different configurations and 
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evaluate if any modification to their original scenario makes improvements. The main 
focus of the experiment is to observe the participants taking the correct steps to protect 
the assets assigned, not the statistical results. Terrorists can successfully attack an asset 
even if proper employment of assets is observed. 

As an exemplar, no analysis was performed. Other potential useful statistics would be 
time to detect all targets or the amount of detections per target. 
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Replication Report 
Entity: Rhib 

Property: Rhib2Detections 
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|Run# 

Count 

Min 

Max 

Mean 

StdDev 

Variance 

1 

14.000 

0.000 

13.000 

6.500 

4.183 

17.500 

2 

4.000 

0.000 

3.000 

1.500 

1.291 

1.667 

3 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

4 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

5 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

6 

4.000 

0.000 

3.000 

1.500 

1.291 

1.667 

7 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

8 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

9 

4.000 

0.000 

3.000 

1.500 

1.291 

1.667 

10 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

11 

4.000 

0.000 

3.000 

1.500 

1.291 

1.667 

12 

4.000 

0.000 

3.000 

1.500 

1.291 

1.667 

13 

14.000 

0.000 

13.000 

6.500 

4.183 

17.500 

14 

7.000 

0.000 

6.000 

3.000 

2.160 

4.667 
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15 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

16 

16.000 

0.000 

15.000 

7.500 

4.761 

22.667 

17 

3.000 

0.000 

2.000 

1.000 

1.000 

1.000 

18 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

19 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

20 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

21 

18.000 

0.000 

17.000 

8.500 

5.339 

28.500 

22 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

23 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

24 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

25 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

26 

4.000 

0.000 

3.000 

1.500 

1.291 

1.667 

27 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

28 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

29 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

30 

1.000 

0.000 

0.000 

0.000 

0.000 

0.000 

31 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

32 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

33 

7.000 

0.000 

6.000 

3.000 

2.160 

4.667 

34 

7.000 

0.000 

6.000 

3.000 

2.160 

4.667 

35 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

36 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

37 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

38 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

39 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

40 

7.000 

0.000 

6.000 

3.000 

2.160 

4.667 

41 

3.000 

0.000 

2.000 

1.000 

1.000 

1.000 

42 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

43 

7.000 

0.000 

6.000 

3.000 

2.160 

4.667 

44 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 

45 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

46 

5.000 

0.000 

4.000 

2.000 

1.581 

2.500 

47 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

48 

6.000 

0.000 

5.000 

2.500 

1.871 

3.500 
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49 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

50 

14.000 

0.000 

13.000 

6.500 

4.183 

17.500 


Replication Report 
Entity: Rhib 

Property: Rhib2Detections 



>4 »• 97 9* »• 4Q 41 42 4S 44 4f 4* 47 4« 49 90 01 92 f> 

Tot9t<0«tKl»on« 


I TotgoOoOHOont 


Run# 

Count 

Min 

Max 

Mean 

StdDev 

Variance 

1 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

2 

12.000 

0.000 

11.000 

5.500 

3.606 

13.000 

3 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

4 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

5 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

6 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

7 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

8 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

9 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

10 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

11 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

12 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 
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13 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

14 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

15 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

16 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

17 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

18 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

19 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

20 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

21 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

22 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

23 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

24 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

25 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

26 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

27 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

28 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

29 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

30 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

31 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

32 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

33 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

34 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

35 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

36 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

37 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

38 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

39 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

40 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

41 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

42 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 

43 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

44 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 

45 

10.000 

0.000 

9.000 

4.500 

3.028 

9.167 

46 

11.000 

0.000 

10.000 

5.000 

3.317 

11.000 
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47 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

48 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

49 

9.000 

0.000 

8.000 

4.000 

2.739 

7.500 

50 

8.000 

0.000 

7.000 

3.500 

2.449 

6.000 


Summary Report 


Entity 

Property 

Count 

Min 

Max 

Mean 

StdDev 

Variance 

Rhib 

Rhib 1 Detections 

2.000 

0.000 

1.000 

0.500 

0.707 

0.500 

Rhib 

Rhib2Detections 

2.000 

0.000 

11.294 

5.647 

7.986 

63.782 


Conclusions and Recommendations 

Conclusions. Participant number 20 completed the simulation assessment portion with 
83% success rate. 

Recommendations for Future Work. Based on the results the subject matter expert 
recommends that participant number 20 reviews the following subjects: how to identify 
vulnerabilities and threat assessment portion of the lesson. 


This report was autogenerated by Viskit modeling tool using the Simkit discrete- 
event simulation (DES) libraries, online at https ;//diana.nps.edu/Simkit 
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APPENDIX O. SUPPORTING MATERIALS 


A. AT/FP LESSON DVD CONTENTS 

The AT/FP lesson DVD is organized as follows: 

CBT_And_Simulation lesson folder 
Data folder 
Narration folder 
Player folder 
Lesson player 
CBT_VS_CI lesson folder 
Data folder 
Narration folder 
Player folder 
Lesson player 

B. AT/FP LESSON DVD REQUESTS 

The DVD that contains the CBT lessons created for this thesis can be requested 
via mail from the Savage Research Group. 

Naval Postgraduate School, MOVES Institute, Monterey, California. 

Terry Norbraten or Don Brutzman 
1 University Circle, Monterey Ca. 93943. 
e-mail requests can be made to: 

Terry Norbraten tdnorbra @ nps. edu 
Don Brutzman Brutzman @ nps. edu 
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